Selective Coupling of Decoupled Informative Regions: Masked Attention Alignment for Data-Free Quantization of Vision Transformers

Technical Summary: Masked Attention Alignment for Data-Free Quantization of Vision Transformers

Problem Statement
Data-Free Quantization (DFQ) aims to compress pre-trained full-precision Vision Transformers (ViTs) into quantized models without access to original training data, addressing critical data privacy and security concerns. While DFQ has shown success with Convolutional Neural Networks (CNNs) by leveraging Batch Normalization statistics, its migration to ViTs faces significant challenges. ViTs rely on Layer Normalization and self-attention mechanisms, lacking the stable distributional priors found in CNNs. Consequently, existing DFQ methods for ViTs often generate synthetic samples that suffer from semantic dispersion (where semantics are scattered across the entire image rather than forming coherent structures) and attentional disparity (where the quantized model $Q$ fails to align its attention maps with the full-precision model $P$, particularly due to the absence of discriminative regions). These issues lead to suboptimal calibration and performance degradation, especially in ultra-low precision settings (e.g., 3-bit).

Methodology: MaskAQ
The authors propose MaskAQ, a novel approach that reframes DFQ for ViTs through an Information Bottleneck (IB) perspective. The core insight is that self-attention semantics are not uniformly distributed but are concentrated in a sparse subset of patches termed Informative Regions (IR). MaskAQ aims to decouple these IRs from noisy backgrounds and align them with the quantized model $Q$. The method consists of three primary components:

1. Informative Region Decoupling:
   To address semantic dispersion, MaskAQ separates informative patches from the background. It defines IRs based on attention weights and employs a differential entropy maximization strategy over patch similarity. By maximizing the entropy of the similarity distribution between attention vectors, the method encourages diverse attention patterns, preventing attention collapse and ensuring a clear separation between semantically critical patches and noisy backgrounds. This is formulated as a regularizer ($L_{fb}$) that promotes structural coherence in synthetic samples.

2. Masked Attention Coupling:
   To address attentional disparity, MaskAQ introduces an adaptive masking mechanism. Instead of aligning full attention maps (which may over-regularize irrelevant background), the method constructs a binary mask to select a strict subset of informative patches. A masked attention alignment objective ($L_{align}$) is then used to minimize the difference between the attention maps of $P$ and $Q$ only within these selected regions. This ensures that the synthetic samples focus on the specific semantic regions that $Q$ needs to learn, bridging the gap between the two models without being misled by quantization-induced noise in non-informative areas.

3. Periodic Sample Refreshing:
   Recognizing that the state of the quantized model $Q$ evolves during the calibration process, the initial synthetic samples may become misaligned with $Q$'s current capabilities. MaskAQ employs a periodic refreshing strategy, where synthetic samples are regenerated at fixed intervals throughout training. This allows the method to continually adapt to the evolving state of $Q$, preserving the desirable mutual information between the samples and the model's outputs.

Key Contributions

• Identification of Critical Bottlenecks: The paper explicitly identifies semantic dispersion and attentional disparity as the primary reasons for the performance gap in existing ViT DFQ methods, particularly under ultra-low precision.
• Information Bottleneck Perspective: The authors shift the DFQ goal from merely approximating real data distributions to preserving the crucial mutual information required for calibrating $Q$. They provide theoretical guarantees (Theorems 1 and 2) demonstrating that aligning $P$ and $Q$ on informative regions is sufficient to maintain predictive power.
• Novel Architecture (MaskAQ): The integration of differential entropy-based region decoupling, adaptive masked attention coupling, and periodic refreshing creates a robust pipeline for high-quality sample synthesis.
• Comprehensive Evaluation: The method is validated across multiple backbones (ViT, DeiT, Swin Transformer) and downstream tasks (classification, object detection, semantic segmentation).

Experimental Results
Extensive experiments on the ImageNet dataset demonstrate that MaskAQ outperforms state-of-the-art approaches (including MimiQ, PSAQ-ViT V2, and CLAMP-ViT) across various bit widths.

• Classification: Under 3-bit quantization (3w3a), MaskAQ achieves up to 3.1% Top-1 accuracy gain over MimiQ on DeiT-T. In 4-bit settings, it consistently improves accuracy by margins ranging from 1.4% to over 7% compared to competitors.
• Downstream Tasks: The method also shows superior performance on object detection (COCO) and semantic segmentation (ADE20K) tasks, confirming its generalizability beyond classification.
• Ablation Studies: Removing any of the three core components (entropy regularizer, masked alignment, or sample refreshing) leads to significant performance drops, validating the necessity of each module.

Significance
The paper claims that MaskAQ represents a significant advancement in the field of Data-Free Quantization for Vision Transformers. By moving beyond simple distribution matching to focus on selective coupling of decoupled informative regions, the method effectively mitigates the unique challenges posed by ViT architectures. The ability to maintain high accuracy in ultra-low precision scenarios (3-bit) without access to real data is highlighted as a crucial step toward deploying secure, efficient ViTs on resource-constrained edge devices. The work underscores that preserving the specific mutual information between synthetic samples and the quantized model's output is more critical than generating visually perfect images.