💻 computer science

Steerable Visual Representations

This paper introduces "Steerable Visual Representations," a novel approach that injects natural language prompts directly into the layers of pretrained Vision Transformers via early fusion, enabling the guidance of visual features toward specific concepts while maintaining high performance on generic visual tasks and achieving zero-shot generalization.

Original authors: Jona Ruthardt, Manu Gaur, Deva Ramanan, Makarand Tapaswi, Yuki M. Asano

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jona Ruthardt, Manu Gaur, Deva Ramanan, Makarand Tapaswi, Yuki M. Asano

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, super-observant security guard named DINO. DINO is trained to look at a room and immediately shout out the most obvious thing he sees. If you show him a photo of a living room with a cat sleeping on a sofa, a bookshelf in the corner, and a remote control on the table, DINO will only care about the cat. He ignores everything else because the cat is the "star of the show."

This is how most current AI vision models work. They are great at seeing the main subject, but they are terrible at being told, "Actually, ignore the cat; I want to know about the bookshelf."

Enter SteerViT (Steerable Visual Representations).

The researchers behind this paper created a new system that turns DINO into a compliant tour guide. Now, you can whisper a command to the AI, like "Look at the bookshelf," and suddenly, the AI's entire focus shifts. It stops caring about the cat and starts analyzing the bookshelf in detail. If you then say, "Look at the remote," it instantly pivots again.

Here is how they did it, using some simple analogies:

1. The Problem: The "Salty" Camera

Think of standard AI vision models like a camera with a fixed, sticky focus. No matter what you ask it to look at, it always zooms in on the biggest, brightest, or most colorful object in the frame. It's like a camera that refuses to focus on anything other than the main actor in a movie, even if the director yells, "Focus on the background prop!"

2. The Solution: The "Language Remote Control"

The authors built a remote control for the AI's brain.

Old Way (Late Fusion): Imagine trying to change the camera's focus after it has already taken the picture. You can add a filter or write a note on the photo, but the picture itself is already blurry regarding the background. This is how older models (like CLIP) work; they look at the image, then look at the text, and try to mash them together at the very end.
New Way (SteerViT / Early Fusion): Instead, SteerViT puts the remote control inside the camera lens itself. As the camera is taking the picture, you are whispering instructions into its ear. "Focus on the bookshelf." The camera adjusts its internal gears while it sees the light, ensuring the bookshelf is sharp and the cat is blurry.

3. How It Works: The "Lightweight Adapter"

You might think, "To make a camera this smart, you probably need to rebuild the whole thing with a massive new engine."

The Reality: The researchers didn't rebuild the engine. They took a high-performance, pre-built engine (a powerful AI called DINOv2) and installed a tiny, 21-million-parameter "adapter" (about the size of a small app) inside it.
The Mechanism: They added a "gated cross-attention" layer. Think of this as a traffic cop inside the AI's brain. When the AI is processing an image, the traffic cop listens to your text prompt. If you say "bookshelf," the cop directs the AI's attention to the bookshelf pixels and tells the cat pixels to "stand down."

4. Why It's a Big Deal: The "Swiss Army Knife" Effect

Usually, in AI, you have to choose between being smart or being flexible.

Specialized Models: Some models are great at finding specific things (like a metal detector finding a coin) but are useless for anything else.
General Models: Some models are great at general tasks but can't be told what to look for.
SteerViT: This is the Swiss Army Knife. It keeps the original model's super-smart vision (it's still great at recognizing cats, cars, and textures) but adds the ability to be steered by text. It gets the best of both worlds without needing to be retrained from scratch.

5. Real-World Magic: Zero-Shot Generalization

The coolest part is that this works on things the AI has never seen before, without any extra training.

Example: Imagine you train the AI to look for "anomalies" (defects) in factory parts. You tell it, "Find the scratch on this metal nut."
The Result: Even if the AI has never seen a metal nut before, because it understands the concept of "scratch" and "metal" from your text, it can find the defect immediately. It's like giving a chef a recipe for a dish they've never made, and they somehow make it perfectly because they understand the principles of cooking.

Summary

SteerViT is like giving a super-intelligent but stubborn camera a voice command. Instead of the camera deciding what is important, you decide. You can tell it to ignore the main subject and focus on the tiny details, or to group things by color, shape, or texture, all by simply typing a sentence. It makes AI vision much more like human vision, where we can choose to focus on whatever we want, rather than just staring at the loudest thing in the room.

1. Problem Statement

Pretrained Vision Transformers (ViTs) like DINOv2 and MAE provide high-quality, generic visual features suitable for tasks like classification and segmentation. However, they suffer from saliency bias: they naturally focus on the most prominent objects in an image (e.g., a cat in a room) and ignore less salient but relevant concepts (e.g., a remote control or bookshelf).

Existing solutions have limitations:

Unimodal ViTs: Cannot be steered by text; they are query-agnostic.
Multimodal LLMs (MLLMs): Can be guided by text but often produce language-centric representations that lose visual fidelity. They also require billions of parameters and late fusion (text interacts only after visual encoding), meaning text does not influence the early visual processing stages.
Open-Vocabulary (OV) Localization: Models like SAM3 are steerable but produce features optimized specifically for localization, lacking generalizability for other downstream tasks.

The authors argue for a new class of representations that satisfy three desiderata:

Steerability: The ability to adapt features to specific text prompts (focusing on non-salient objects).
Representation Quality: Preserving the high performance of the underlying ViT on generic visual tasks.
Early Fusion: Integrating language directly into the visual encoding process (early layers) rather than just at the output.

2. Methodology: SteerViT

SteerViT introduces a framework that conditions a frozen pretrained ViT on natural language via early fusion using lightweight cross-attention.

Architecture

Frozen Encoders: The visual encoder (e.g., DINOv2, SigLIP, MAE) and the text encoder (RoBERTa-Large) remain frozen.
Multimodal Adapter: Text embeddings are projected into the visual embedding space via a trainable two-layer MLP.
Gated Cross-Attention (CA): The core innovation. The authors insert trainable cross-attention layers into the residual stream of the ViT (specifically, every other block).
- Mechanism: Visual patch tokens act as queries, while the adapted text tokens act as keys and values. This allows the visual encoder to "attend" to the text prompt during feature extraction.
- Gating: A learnable scalar $\alpha_\ell$ with a $\tanh$ gate controls the strength of the text injection. Initialized to zero, the model starts as a standard ViT and gradually learns to incorporate text cues.
Parameter Efficiency: Only the cross-attention layers and the MLP projector are trainable, adding approximately 21M parameters (two orders of magnitude fewer than MLLMs).

Training Objective

The model is trained using a referential segmentation pretext task:

Input: An image and a text prompt describing a target object.
Task: Predict which image patches correspond to the referred object.
Loss: Soft cross-entropy loss on patch-level segmentation masks. This forces the cross-attention layers to route textual information to the specific visual patches corresponding to the prompt, effectively "steering" the feature space.

3. Key Contributions

SteerViT Framework: A method to equip any pretrained ViT with text-steerable capabilities by interleaving lightweight cross-attention layers, achieving Pareto improvement (high steerability without sacrificing representation quality).
Early Fusion Paradigm: Demonstrates that conditioning the visual encoder during feature extraction (early fusion) is superior to late fusion for controlling visual attention and semantics.
Zero-Shot Generalization: The method enables zero-shot transfer to novel domains (e.g., industrial anomaly detection, personalized object discrimination) without task-specific fine-tuning.
New Benchmarks: Introduction of CORE (COnditional REtrieval) to measure steerability and MOSAIC to evaluate targeted attention routing.

4. Experimental Results

The paper evaluates SteerViT against Unimodal ViTs, Cross-modal encoders (CLIP/SigLIP), MLLMs (InternVL, Qwen), and OV localization models (SAM3, GroundingDINO).

Steerability (CORE Benchmark):
- SteerViT achieves 96.0% top-1 retrieval accuracy on text-guided image retrieval, significantly outperforming DINOv2 (43.7%) and MLLMs (e.g., InternVL3-2B at ~70%).
- It successfully shifts focus from scene-level concepts to specific, non-salient objects (e.g., finding a "fruit bowl" in a kitchen scene).
- Ablation: Late fusion (adding text after the ViT) fails to steer frozen features effectively, confirming the necessity of early fusion.
Representation Quality:
- Unlike OV models which degrade on generic tasks, SteerViT preserves the underlying ViT's quality.
- On fine-grained classification (ImageWoof, StanfordCars) and segmentation (ADE20k), SteerViT matches or slightly exceeds the base ViT performance.
- Control Knob: The gating mechanism allows continuous interpolation between the vanilla ViT and the steered state. A scaling factor of $\omega=0.6$ often yields the optimal trade-off.
Granularity and Topology:
- PODS Benchmark: SteerViT can control the granularity of features. Coarse prompts ("mug") yield category-level clustering, while detailed prompts ("white mug with black dots") enable instance-level discrimination, outperforming fine-tuned DINOv2 variants.
- Embedding Space: Text conditioning can reorganize the embedding topology, grouping images by semantic hierarchy (e.g., "animals") or compositional attributes (e.g., "has eyes"), regardless of object class.
Zero-Shot Domain Transfer:
- Anomaly Detection: On the MVTec AD dataset, SteerViT achieves 82.1 PRO (Per-Region-Overlap), matching dedicated anomaly detection methods (FADE: 84.5) and outperforming general segmentation models (SAM3: 54.5) without task-specific training.

5. Significance

SteerViT proposes a paradigm shift in multimodal learning: conditioning vision on language rather than the traditional approach of conditioning language on vision (as in MLLMs).

Efficiency: It achieves state-of-the-art steerability with only ~21M trainable parameters, making it highly efficient compared to billion-parameter MLLMs.
Flexibility: It unlocks the latent capabilities of frozen, high-quality vision encoders, allowing them to adapt to specific tasks (anomaly detection, personalized retrieval) via natural language prompts without retraining the backbone.
Insight: The work validates that early fusion of text into visual processing is crucial for mimicking human top-down attention mechanisms, where prior knowledge (text) guides visual parsing.

In summary, SteerViT demonstrates that visual representations can be dynamically steered to focus on any desired concept while maintaining the robustness and generalizability of the original foundation model, offering a powerful, parameter-efficient tool for next-generation vision systems.