💻 computer science

LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision

The paper introduces LookWhere, a self-supervised method that efficiently handles high-resolution visual recognition by jointly learning to select relevant image regions and extract features through a low-resolution selector and high-resolution extractor, achieving significant reductions in computational cost while maintaining or improving accuracy across various tasks.

Original authors: Anthony Fuller, Yousef Yassin, Junfeng Wen, Daniel G. Kyrollos, Tarek Ibrahim, James R. Green, Evan Shelhamer

Published 2026-02-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Anthony Fuller, Yousef Yassin, Junfeng Wen, Daniel G. Kyrollos, Tarek Ibrahim, James R. Green, Evan Shelhamer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to recognize a friend in a massive, high-resolution crowd photo. A traditional computer vision model is like a person who insists on examining every single face in that photo, one by one, to find your friend. If the photo is huge (like a 4K image), this takes forever and requires a lot of brainpower.

The paper "LookWhere" proposes a smarter, more efficient way to do this. Instead of looking at everything, the system learns where to look and what to see, skipping the boring parts entirely.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Whole Photo" Trap

Current AI models (called Vision Transformers) are getting incredibly smart, but they are also getting huge and expensive to run. When you feed them a high-resolution image, they break it into thousands of tiny pieces (tokens) and analyze every single one.

The Analogy: Imagine trying to find a specific book in a library by reading the cover of every single book on every shelf, even the ones clearly labeled "Cooking" when you are looking for "History." It's a waste of time.

2. The Solution: A Two-Person Team

The authors created a system called LookWhere that splits the job into two specialized roles, working together like a detective and a specialist.

The Selector (The "Quick Glance"):
- What it does: This part looks at a tiny, blurry, low-resolution version of the image. It's like squinting at the photo from far away.
- The Job: It quickly decides, "Hey, the interesting stuff is over there in the top right corner," and ignores the rest. It draws a map of where to focus.
- The Benefit: Because it only looks at a small, blurry version, it is incredibly fast and cheap to run.
The Extractor (The "Close-Up"):
- What it does: This part is the heavy lifter. It only looks at the specific, high-resolution patches (the "interesting" spots) that the Selector pointed to.
- The Job: It examines those few spots in high detail to figure out exactly what is there (e.g., "That's a red traffic sign" or "That's a sparrow").
- The Benefit: It never wastes time looking at the empty sky or the blurry background.

3. How They Learn: The "Mentor" System

You might wonder: How does the "Quick Glance" person know where to look without seeing the whole picture first?

They use a Self-Supervised Mentor.

The Mentor: The researchers used a very powerful, pre-trained AI (called DINOv2) that has already "seen" the internet. This Mentor is smart enough to know what parts of an image are interesting, even without being told what the task is.
The Lesson: The Mentor looks at the full, high-resolution image and says, "I am looking at this specific patch because it has important details."
The Training: The "Quick Glance" (Selector) and the "Close-Up" (Extractor) try to mimic the Mentor's behavior.
- The Selector learns to guess where the Mentor is looking, just by seeing the blurry version.
- The Extractor learns to guess what the Mentor sees, just by looking at the few patches the Selector picked.
The Result: The team learns to ignore the boring parts and focus on the important ones, all without needing a human to tell them what to look for.

4. The Results: Fast and Accurate

The paper tested this on several tasks:

Traffic Signs: In a high-resolution photo of a street, the system found the signs 6 times faster and used 34 times less computing power than standard methods, while keeping the same accuracy.
Birds and Billiards: It worked just as well on identifying specific bird species or the position of billiard balls, proving it can handle fine details without looking at the whole background.
General Tasks: Even on standard tasks like identifying objects in general photos (ImageNet) or segmenting scenes (ADE20K), it was faster and often more accurate than other "adaptive" methods.

The Bottom Line

LookWhere is like hiring a smart assistant who knows exactly where to zoom in on a photo. Instead of forcing a computer to stare at every pixel in a massive image, it learns to skip the empty space and focus only on the action. This makes AI faster, cheaper to run, and just as accurate as the old, slow way of doing things.

Key Takeaway: The system doesn't just "prune" (cut out) parts of an image after it's already looked at them; it decides before it starts looking which parts are worth the effort.

Technical Summary: LookWhere

Problem Statement

Vision Transformers (ViTs) are becoming increasingly large and accurate, but their computational cost scales quadratically with image resolution due to the quadratic growth of tokens. This expense is prohibitive for high-resolution inputs common in photography, remote sensing, autonomous driving, and medical imaging. Existing adaptive computation methods face significant limitations:

Token Reduction: Methods that prune or merge tokens (e.g., PiToMe, DTEM) still require processing the full set of input tokens in early layers, incurring high initial costs.
Token Selection: Methods that select tokens (e.g., DPS, IPS) often rely on complex, expensive per-task optimization or discrete optimization techniques that are difficult to train and transfer.
Efficiency Gap: There is a lack of methods that can efficiently select where to look and extract what to see from high-resolution inputs without ever processing the full image, while maintaining transferability across diverse tasks.

Methodology: LookWhere

The authors propose LookWhere, a framework for self-supervised adaptive computation that factorizes the problem into "where to look" and "what to see" using a paired Selector-Extractor architecture.

Core Architecture

Selector (Where to Look):
- Operates on a low-resolution version of the input image.
- A shallow ViT (truncated depth) processes this low-res input to generate a "selector map" (a high-resolution attention map) predicting the locations of informative patches.
- It also generates low-resolution global tokens (class and register tokens) to be reused by the extractor.
Extractor (What to See):
- Operates on a sparse set of high-resolution patches selected by the selector (top- $k$ patches).
- It tokenizes only these selected high-res patches.
- It re-uses the global tokens generated by the selector (replacing its own learnable tokens) to avoid redundant computation.
- It processes these sparse inputs to produce a full representation of the high-resolution image, which is then used for task-specific prediction.

Training: What-Where Distillation

The system is pretrained using self-supervised distillation from a large, frozen teacher model (DINOv2). This allows the system to learn without task-specific labels.

Teacher: A self-supervised ViT (DINOv2) that processes the full high-resolution input.
Distillation Targets:
- Where: The selector learns to predict the teacher's attention map (indicating salient regions) using only the low-resolution input.
- What: The extractor learns to predict the teacher's class and patch token representations (the semantic content) using only the sparse high-resolution patches and the selector's low-res global tokens.
Loss Function: A weighted sum of three losses:
1. Class Token Distillation (MSE): Aligns global representations.
2. Patch Token Distillation (MSE): Aligns local representations.
3. Attention Distillation (KL Divergence): Aligns the selector's predicted map with the teacher's attention map.
Efficiency: The teacher processes the full image once to generate targets. The student (Selector + Extractor) never processes the full high-resolution image during training or inference, making pretraining and inference highly efficient.

Fine-tuning

For downstream tasks, only the extractor and a task-specific predictor (e.g., a linear layer) are fine-tuned using supervised learning. The selector remains frozen, allowing the "where to look" strategy to generalize across tasks.

Key Contributions

Novel Architecture: A selector-extractor framework that decouples location selection (low-res) from feature extraction (sparse high-res), avoiding full-resolution processing entirely.
Self-Supervised Pretraining: A "what-where distillation" scheme that jointly trains the selector and extractor from a self-supervised teacher, enabling transfer across tasks without per-task optimization for selection.
Hardware Efficiency: Unlike prior token selection methods requiring complex discrete optimization or clustering, LookWhere relies on standard, accelerated GPU operations (standard ViT layers), making it faster on current hardware.
High-Resolution Efficiency: Demonstrates significant gains in high-resolution scenarios where quadratic scaling is most problematic.

Experimental Results

The paper evaluates LookWhere on standard benchmarks (ImageNet, ADE20K) and high-resolution benchmarks (Traffic Signs, Birds, Billiard Balls).

ImageNet Classification: LookWhere achieves state-of-the-art (SoTA) accuracy/efficiency trade-offs. For ViT-S, it is 1.36× faster than the second-best method (DTEM) while maintaining comparable accuracy.
ADE20K Segmentation: LookWhere outperforms DTEM in accuracy while being ≥2× faster at comparable computation levels.
High-Resolution Tasks (Traffic Signs):
- LookWhere reduces FLOPs by up to 34× and inference time by 6× compared to baselines like IPS, while maintaining competitive accuracy.
- It outperforms its teacher (DINOv2) in efficiency while matching or exceeding its accuracy on sparse recognition tasks.
Fine-Grained Recognition (Birds): Achieves nearly equal accuracy to DINOv2 while processing only 10% of the patches, resulting in significant savings in speed, memory, and FLOPs.
Training Efficiency: LookWhere requires significantly less cumulative FLOPs during fine-tuning compared to other adaptive methods (e.g., rivaling DPS in accuracy with a fraction of the training cost).
Robustness: Preliminary tests show LookWhere is more robust to small $\epsilon$ adversarial attacks than DINOv2, suggesting sparse computation may offer inherent robustness benefits.

Significance and Claims

The paper claims that LookWhere represents a shift in adaptive computation by:

Eliminating Full-Resolution Processing: Unlike token reduction methods, it never processes the full input, offering a fundamental efficiency advantage at high resolutions.
Simplicity and Transferability: Unlike prior selection methods requiring complex per-task optimization, LookWhere uses a simple, unified pretraining strategy that transfers effectively across global (classification) and local (segmentation) tasks.
Practicality: It leverages standard transformer operations, making it compatible with current hardware accelerators without the overhead of custom clustering or discrete optimization algorithms.

The authors position LookWhere as a solution to the "quadratic cost" bottleneck of ViTs, enabling efficient, accurate recognition on high-resolution imagery by learning to ignore irrelevant pixels through self-supervision. They acknowledge limitations, such as the current lack of selector fine-tuning and the spatial-only nature of the method, suggesting future work could explore temporal (video) or spectral (remote sensing) adaptations.