What DINO saw: ALiBi positional encoding reduces positional bias in Vision Transformers

This paper demonstrates that finetuning Vision Transformers with ALiBi positional encoding effectively reduces inherent positional biases while preserving general semantic representations, thereby enabling successful zero-shot adaptation and segmentation in direction-independent domains like material science microscopy.

The Gist

The Big Picture: The "Blind Spot" of AI Eyes

Imagine you have a super-smart robot named DINO (specifically, a model called DINOv2). DINO is a master at looking at pictures and understanding what's in them. If you show it a photo of a dog, it knows it's a dog. If you show it a car, it knows it's a car. It's like a brilliant art critic who can describe a painting perfectly.

However, this paper discovered that DINO has a weird blind spot.

When DINO looks at a picture, it doesn't just see the objects; it also secretly sees the location of the objects in a very rigid way. It's like DINO has a mental map that says, "Everything on the left side of the image is slightly different from everything on the right side," even if the image is just a blank wall or a random texture.

This is a problem for scientists who study materials (like the inside of a battery or a piece of metal). These images often look like uniform, grainy textures with no "left" or "right" preference. Because DINO is obsessed with position, it gets confused. It tries to force a "left-to-right" pattern onto a picture that has none, leading to messy, incorrect results when scientists try to use it to analyze these materials.

The Problem: The "Ruler" in the Brain

The authors found that DINO's brain (its neural network) has a built-in ruler.

• How it works: When DINO was trained, it learned to associate specific parts of its brain with specific spots on the image (top-left, bottom-right, etc.).
• The Glitch: Even when looking at a picture of pure white noise or a uniform metal surface, DINO's brain lights up in a "ramp" pattern. It thinks, "Oh, the pixels on the left are 'low,' and the pixels on the right are 'high'."
• The Consequence: When scientists tried to use DINO to segment (outline) different parts of a battery image, the AI would draw lines based on where the pixels were, not what they were. It was like a painter who refuses to paint a blue sky unless the paint is on the left side of the canvas.

The Solution: The "ALiBi" Fix

The team decided to fix DINO's brain. They didn't want to throw DINO away because it was so smart at recognizing objects. They just wanted to remove that annoying "ruler."

They replaced DINO's original "learned ruler" with a new system called ALiBi (Attention with Linear Biases).

The Analogy: The "Relative Distance" Game

• Old DINO (The Absolute Ruler): Imagine a student who memorizes that "Question 1 is always on the left page" and "Question 2 is always on the right page." If you give them a test with the pages shuffled, they get confused.
• New DINO (The Relative Distance): Imagine a student who doesn't care about the page number. They only care about, "How far away is this question from the one I'm looking at right now?"

By switching to this "Relative Distance" system (ALiBi), the model stops caring about absolute coordinates (Top-Left vs. Bottom-Right) and starts caring only about how close things are to each other.

The Experiment: Cleaning Up the Vision

The researchers took a pre-trained DINO model, ripped out its old "ruler," and installed the new "ALiBi" system. Then, they taught it to look at the same pictures again, but this time, they told it: "Don't worry about the position; just tell me what the object is."

The Results:

1. The "Ruler" Disappeared: When they tested the new model on uniform images (like metal grains or white noise), the weird "left-to-right" patterns vanished. The model became "homogenous"—it treated the left side and the right side exactly the same.
2. It Still Knew What Things Were: Surprisingly, the model didn't lose its intelligence. It could still recognize dogs, cars, and complex battery structures just as well as the old version.
3. Better Segmentation: When they used this new model to help scientists slice up images of batteries, the results were perfect. The AI stopped drawing lines based on position and started drawing lines based on the actual material (like distinguishing a pore from a solid particle).

Why This Matters

This is a big deal for Materials Science.

Scientists often look at microscopic images of batteries, metals, or rocks. These images are often huge, grayscale, and look very similar everywhere (no clear "subject" like a dog or a car).

• Before: They had to use complex workarounds or get frustrated because the AI kept getting confused by the image's position.
• Now: They have a "clean" AI that looks at the material itself, not the map coordinates. This allows for better analysis of battery lifespans, stronger metals, and better manufacturing.

Summary in One Sentence

The authors took a super-smart AI that was accidentally biased by its own internal map, gave it a new "relative distance" brain, and turned it into a perfectly balanced tool that can finally analyze uniform scientific images without getting confused.

Technical Summary

1. Problem Statement

Vision Transformers (ViTs), particularly feature foundation models like DINOv2, have become powerful tools for downstream tasks such as segmentation and clustering. However, the authors identify a critical flaw: these models exhibit strong positional biases.

• The Issue: The learned positional encodings (PE) in standard ViTs cause the output features to contain artifacts that correlate with the token's position in the image (e.g., left-to-right or top-to-bottom gradients) rather than just semantic content.
• The Consequence: This bias is particularly detrimental in materials science and microscopy. Unlike natural images (which have distinct objects and backgrounds), materials science images (e.g., SEM/TEM cross-sections) are often homogeneous, grayscale, and lack a preferred direction.
• The Failure Mode: When using DINOv2 features for zero-shot or trainable segmentation on these homogeneous images, the model's positional bias leads to poor segmentation results, where the classifier learns to segment based on image location rather than material properties (e.g., failing to distinguish pores from binder in a battery electrode).

2. Methodology

The authors propose a two-stage approach: characterizing the bias and then mitigating it via architectural modification and fine-tuning.

A. Characterization via Linear Probing

To quantify the extent of positional bias, the authors employed linear probing:

• Setup: They trained linear regressors to map ViT output features (both full stacks and individual channels) to 1D ramp functions (left-right, up-down, diagonal, radial) and coordinate maps $(x, y)$.
• Datasets: They tested on homogeneous microscopy images, natural textures (DTD), and random noise.
• Metric: The coefficient of determination ($R^2$) was used to measure how well the features predicted position. High $R^2$ indicates strong positional bias.
• Findings:
  • DINO and DINOv2 models showed very high $R^2$ scores (e.g., 0.83 for DINOv2-S on micrographs), indicating that specific feature channels act as "positional ramps."
  • Even models using Rotary Positional Encodings (RoPE) (like DINOv3) exhibited strong bias, with bias increasing through network layers.
  • Supervised models (e.g., ViT trained on ImageNet) showed negligible bias, suggesting the issue is specific to self-supervised learning objectives combined with learned PEs.

B. Proposed Solution: ALiBi-Dv2

The authors developed a method to "de-bias" a pre-trained DINOv2 model without retraining from scratch.

1. Architecture Change: They replaced the standard learned 1D positional encoding with ALiBi (Attention with Linear Biases).
   • ALiBi encodes position as a linear bias added to attention scores based on the relative distance between tokens.
   • They implemented a 2D-aware ALiBi with cylindrical boundary conditions (wrapping) to ensure symmetry and avoid edge effects.
   • The distance metric was normalized to [0, 1].
2. Training Strategy (Fine-tuning):
   • Teacher-Student Setup: They used the original, biased DINOv2 embeddings as the training target (teacher).
   • Rationale: The original embeddings contain the desired rich semantic information. By forcing the new ALiBi model to mimic these embeddings, the model learns to preserve semantics while the ALiBi architecture inherently prevents the re-encoding of positional ramps.
   • Channel Zeroing: During training, the four most positional channels (identified via probing) were zeroed out to accelerate convergence.
   • Multiscale Training: They included a short period of training at higher resolutions (518px) to improve length generalization.

3. Key Contributions

1. Systematic Characterization of Bias: The paper provides empirical evidence that positional bias is a pervasive issue in self-supervised ViTs (DINO, MAE, DINOv3) but is minimal in supervised models. They demonstrate that this bias manifests as simple linear ramps in specific feature channels.
2. ALiBi-Dv2 Model: They successfully fine-tuned a DINOv2 checkpoint to use 2D-aware ALiBi positional encoding, creating a model (ALiBi-Dv2) that retains the semantic richness of DINOv2 but eliminates positional artifacts.
3. Demonstration of "Bias-Free" Semantics: They proved that it is possible to use biased teacher embeddings to train an unbiased student model, provided the architecture (ALiBi) constrains the representation space.
4. Application to Materials Science: They demonstrated the practical utility of this approach in trainable segmentation of complex microscopy images (e.g., battery electrodes, steel alloys), where previous models failed due to positional artifacts.

4. Results

Quantitative Benchmarks

• Linear Probing Scores: The $R^2$ score for predicting position dropped drastically for ALiBi-Dv2 (from 0.83 for DINOv2 to -0.23 on micrographs), indicating the removal of positional ramps.
• Semantic Segmentation (VOC/ADE20K): When used as a frozen backbone for linear probe segmentation on standard benchmarks, ALiBi-Dv2 performed comparably or slightly better than DINOv2 (e.g., mIoU 0.692 vs 0.680 on VOC07). This confirms that removing positional bias did not degrade general semantic understanding.
• Robustness: ALiBi-Dv2 showed improved robustness to image transformations (flips, rotations) compared to DINOv2.

Qualitative Analysis

• PCA Visualizations: Feature PCA visualizations showed that DINOv2 and DVT (Denoising ViT) exhibited clear edge effects and gradients (positional artifacts). In contrast, ALiBi-Dv2 produced smooth, homogeneous feature maps that respected object boundaries without positional gradients.
• k-Means Clustering: On homogeneous images (e.g., eutectic alloys), DINOv2 and DVT clustered based on position (radial or vertical splits). ALiBi-Dv2 clustered based on actual material texture and structure.

Application: Trainable Segmentation

• Case Study: In segmenting lithium-ion battery cathodes (distinguishing pores, binder, and active material), standard DINOv2 features led to segmentation errors where the classifier predicted classes based on image location (e.g., "bottom of image = pore").
• Outcome: ALiBi-Dv2 features enabled the classifier to correctly identify the "pore-back" effect (out-of-plane material) and segment complex microstructures with high fidelity, even with sparse user labels.

5. Significance and Conclusion

This work addresses a fundamental limitation in applying foundation models to scientific imaging. By showing that positional bias is a learnable artifact that can be removed via architectural changes (ALiBi) without sacrificing semantic power, the authors enable the use of "off-the-shelf" ViTs in domains where spatial homogeneity is the norm.

• For Computer Vision: It challenges the assumption that learned positional encodings are always optimal, suggesting that relative encodings (like ALiBi) may be superior for tasks requiring spatial invariance.
• For Materials Science: It provides a robust, low-data pipeline for analyzing microstructures, removing the need for extensive data augmentation or custom denoising networks to correct positional artifacts.
• Future Work: The authors note that while they fine-tuned an existing model, training a foundation model from scratch with ALiBi could further optimize these properties. They also highlight that the bias appears to be a general property of self-supervision rather than a specific flaw in the DINO objective.

Code Availability: The authors have released the code for reproducing results at https://github.com/tldr-group/dino-saw.