Do All Vision Transformers Need Registers? A Cross-Architectural Reassessment

1. Problem Statement

Vision Transformers (ViTs), particularly large-scale models like DINOv2, DeiT3, and OpenCLIP, often suffer from artifacts in their attention and feature maps. These artifacts manifest as "high-norm tokens" (tokens with exceptionally high L2 norms) that disrupt the interpretability of the model.

• The Issue: These high-norm tokens appear as outliers in feature maps and disproportionately high attention scores in attention maps. They are hypothesized to occur in regions of redundant visual information, where the model repurposes tokens for internal global computation rather than local feature extraction.
• The Proposed Solution (by Darcet et al., 2024): The introduction of "registers"—empty input tokens added to the sequence—to serve as dedicated memory for global computation, thereby eliminating artifacts and improving interpretability without sacrificing performance.
• The Gap: While Darcet et al. validated these claims on DINOv2-G, it remained unclear if these phenomena and the efficacy of registers generalize across different architectures (e.g., hierarchical models), model sizes (small vs. large), and training paradigms (self-supervised vs. supervised).

2. Methodology

The authors conducted a rigorous cross-architectural reproduction and extension of Darcet et al.'s work.

• Terminology Clarification: The paper first disentangles terms often used interchangeably in prior work:
  • Artifacts: Anomalous patterns in maps.
  • Outliers/High-norm tokens: Tokens with L2 norms in the top 2%.
  • Key Distinction: High-norm tokens in feature maps do not always correlate with artifacts in attention maps across all architectures.
• Models Evaluated:
  • Standard ViTs: DINO, DINOv2 (S, B, L, G), DeiT3 (S, M, H), OpenCLIP (B, H, G).
  • Hierarchical ViTs (H-ViTs): Pyramid Vision Transformer v2 (PVTv2) and Swin Transformer v2 (Swinv2).
• Datasets: ImageNet-1K (for evaluation), LVD-142M (for DINO/DINOv2 training context), and LAION-2B (for OpenCLIP).
• Experimental Tasks:
  1. Artifact Detection: Visualizing L2 norm distributions and attention maps.
  2. Redundancy Analysis: Measuring cosine similarity between high-norm tokens and their spatial neighbors to test if they reside in redundant regions.
  3. Information Content Tasks:
     • Position Prediction: Can a token predict its grid location? (Tests local info).
     • Pixel Reconstruction: Can a token reconstruct its image patch? (Tests local info).
     • Image Classification: Using specific tokens (Normal vs. Outlier) as the global representation. (Tests global info).
  4. Register Evaluation: Training linear classifiers on DINOv2 models with and without registers to measure accuracy and artifact reduction.

3. Key Contributions & Results

A. Reproduction of Darcet et al. (2024) Findings

The authors confirmed several core claims for large standard ViTs (specifically DINOv2-G):

• Claim 1 (Refined): Artifacts are high-norm tokens, but this equivalence is not universal. In some models, attention artifacts do not correspond to high-norm feature tokens.
• Claim 2 (Refined): High-norm tokens are prevalent in large models (>300M params) but also appear frequently in smaller models (e.g., OpenCLIP-B, DeiT3-M). Model size is not the sole determinant.
• Claim 3: High-norm tokens predominantly emerge in regions of redundant visual information (high cosine similarity to neighbors).
• Claim 4: High-norm tokens encode global context but lack local information. They underperform in position/pixel reconstruction tasks but outperform normal tokens in image classification tasks.
• Claim 5: Adding registers effectively eliminates artifacts in attention maps and marginally improves classification accuracy on ImageNet-1K.

B. Cross-Architectural Generalization (New Findings)

The paper extends these findings to smaller models and Hierarchical ViTs (H-ViTs), revealing critical nuances:

• Small Models: Contrary to the assumption that only large models suffer, smaller models (e.g., OpenCLIP-B) exhibit high-norm tokens, sometimes even more prominently than their larger counterparts.
• Hierarchical Architectures (PVT & Swin):
  • Divergent Behavior: The "division of labor" between normal and outlier tokens is architecture-dependent, not universal.
  • Swin Transformer: Outliers concentrate fine-grained appearance/semantic information (better pixel reconstruction and classification), while normal tokens hold spatial/positional info. This is the inverse of standard ViTs.
  • PVT (Pyramid): Outliers encode coarse positional structure (achieving 100% position prediction accuracy) but lack texture details.
  • Implication: The relationship between token norms and local/global information is dictated by the inductive bias of the architecture (e.g., windowed attention vs. pyramid downsampling).

C. Limitations of Registers

• While registers successfully clean feature maps and remove artifacts, they do not significantly improve classification performance when used as input features for downstream linear classifiers.
• The authors suggest that registers may hold global information that overlaps with the [CLS] token, offering no novel signal for classification tasks.

4. Significance

• Terminological Rigor: The paper corrects the conflation of "artifacts," "outliers," and "high-norm tokens," establishing that these concepts are model-specific and not universally interchangeable.
• Architectural Awareness: It demonstrates that the behavior of Vision Transformers is not monolithic. Strategies for artifact mitigation (like registers) or interpretation must account for specific architectural choices (e.g., hierarchical vs. flat, windowed vs. global attention).
• Reproducibility: By reproducing results on a wider range of models (including those not originally tested by Darcet et al.), the paper provides a more comprehensive understanding of the "global information aggregation" phenomenon in ViTs.
• Open Questions: The study highlights that while registers solve the symptom (artifacts), the root cause of why models repurpose tokens for global computation in redundant regions remains an open research question.

Conclusion

The paper confirms that registers are an effective tool for improving the interpretability of Vision Transformers by removing high-norm artifacts. However, it refutes the universality of the underlying mechanisms, showing that the nature of "outlier" tokens and their information content varies significantly across model sizes and architectural families. Future work must consider these architectural nuances when designing or interpreting ViTs.