💻 computer science

CanViT: Toward Active-Vision Foundation Models

The paper introduces CanViT, the first task- and policy-agnostic Active-Vision Foundation Model (AVFM) that utilizes a novel retinotopic-to-spatiotopic architecture and label-free pretraining to achieve state-of-the-art performance in semantic segmentation and image classification with significantly lower computational costs than existing active vision models.

Original authors: Yohaï-Eliel Berreby, Sabrina Du, Audrey Durand, B. Suresh Krishna

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yohaï-Eliel Berreby, Sabrina Du, Audrey Durand, B. Suresh Krishna

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a massive jigsaw puzzle, but you are blindfolded. You can only lift your hand to look at one tiny piece of the puzzle at a time.

Most computer vision models today are like someone who takes a single, giant photo of the whole puzzle and tries to solve it all at once. This works, but it's slow, uses a lot of energy, and if the puzzle is huge (like a high-resolution city map), the computer gets overwhelmed.

CanViT is a new kind of "AI detective" that solves this problem by mimicking how humans actually look at the world. Instead of staring at the whole picture, it takes a series of quick, focused glimpses.

Here is the story of how CanViT works, broken down into simple concepts:

1. The Two-Part Brain: The "Eyes" and the "Notebook"

CanViT has two distinct parts working together, like a detective with a camera and a notebook.

The Eyes (The Backbone): This is the part that looks at the current "glimpse" (a small crop of the image). It's fast and focused, but it has a short memory. Once it looks away, it forgets the details of that specific spot.
The Notebook (The Canvas): This is CanViT's secret weapon. It's a giant, persistent mental map of the entire scene. As the "Eyes" look at different spots, they write notes into the "Notebook." The Notebook remembers where everything is, even if the Eyes are currently looking somewhere else.

The Analogy: Imagine you are exploring a dark cave with a flashlight (the Eyes). You can only see a few feet in front of you. But you have a map (the Canvas) where you draw what you see. Even when you turn the flashlight away, the map still holds the picture of the whole cave.

2. The Magic Trick: "Scene-Relative" Coordinates

Usually, if you take a photo of a cat's nose up close, and then a photo of the cat's ear from far away, a computer gets confused about where those things are relative to each other.

CanViT uses a special trick called Scene-Relative Rotary Position Embeddings (SR-RoPE).

The Metaphor: Imagine the cave has a permanent grid painted on the floor (like a giant graph paper). No matter where you stand or how much you zoom your flashlight in, you always know exactly which square of the grid you are looking at.
This allows CanViT to stitch together a zoomed-in view of a flower and a zoomed-out view of a garden into one coherent picture without getting lost.

3. The "Canvas Attention" (The Efficient Conversation)

In older models, every time the "Eyes" looked at a new spot, they had to re-read the entire "Notebook" to update it. This is like reading a whole encyclopedia every time you write a single sentence. It's slow and expensive.

CanViT invented Canvas Attention, which is a one-way, efficient conversation.

The Analogy: Instead of reading the whole book, the "Eyes" just ask the "Notebook," "What do you know about this specific corner?" and the Notebook whispers the answer. Then, the Eyes write a quick update back.
The Result: This makes the system incredibly fast. It can handle huge scenes without the computer getting tired (running out of memory or processing power).

4. How It Learned: The "Teacher" Method

CanViT didn't learn by being told "This is a cat" or "This is a dog." That would be like teaching a child to drive by only showing them pictures of cars.

Instead, the researchers used a Passive-to-Active Distillation method.

The Setup: They took a super-smart, passive AI (DINOv3) that had already seen millions of high-resolution photos. This AI is the "Teacher."
The Game: They showed the Teacher a full, high-res photo. Then, they showed CanViT a sequence of random, low-res "glimpses" of that same photo.
The Goal: CanViT had to guess what the whole photo looked like based only on those tiny glimpses. It had to fill in the blanks.
The Outcome: CanViT learned to build a rich, 3D understanding of the world just by looking at pieces of it, without needing any labels or human supervision.

5. Why This Matters: The Results

The paper shows that CanViT is a game-changer for two main reasons:

It's Super Efficient: It can understand a complex scene (like a street with cars, people, and trees) using 19.5 times less computing power than previous "active vision" models. It's like getting a Ferrari's performance on a bicycle's battery.
It's Smarter: Even with just one quick glance, it outperforms the best previous models. If you let it take a few more glimpses (like a human scanning a room), its accuracy skyrockets. It can even handle scenes much larger than what it was trained on.

The Big Picture

Before CanViT, there was a huge gap between "Passive Vision" (staring at a whole photo) and "Active Vision" (looking around like a human). Passive models were smart but inefficient; Active models were efficient but dumb.

CanViT bridges that gap. It proves that you can build an AI that sees the world the way humans do—sequentially, efficiently, and with a working memory—without needing complex, slow training methods. It's a blueprint for the next generation of robots and AI that can navigate the real world without getting overwhelmed.

1. Problem Statement

Active Computer Vision (ACV) aims to mimic biological vision by processing scenes through sequential, localized glimpses rather than processing full frames passively. While theoretically more efficient and biologically plausible, existing ACV models suffer from significant limitations:

Lack of Scalability: They lack general-purpose architectures and scalable pretraining pipelines, often remaining confined to simple tasks like digit recognition or fixed-zoom classification.
Poor Dense Prediction: Most active models cannot perform dense prediction tasks (e.g., semantic segmentation) efficiently. Existing attempts (e.g., AME, AdaGlimpse) rely on post-hoc expansion of encoder outputs, leading to intractable computational costs at high resolutions.
Policy Dependence: Prior work often conflates "how to see" (perception) with "where to look" (policy), relying heavily on Reinforcement Learning (RL) for action selection, which complicates training and deployment.
Gap with Passive Models: Passive Vision Foundation Models (like DINOv3) excel at general-purpose understanding but lack mechanisms for memory and sequential processing. Active models lag significantly behind passive models in accuracy and efficiency.

2. Methodology: CanViT Architecture

The authors introduce CanViT (Canvas Vision Transformer), the first task- and policy-agnostic Active-Vision Foundation Model (AVFM). It decouples perception from action selection.

Core Architecture

CanViT utilizes a dual-stream recurrent architecture:

Backbone Stream (Retinotopic): A standard ViT backbone that processes low-resolution, localized glimpses ( $128^2$ px). It is ephemeral, processing the current view without retaining spatial grid structure across time.
Canvas Stream (Spatiotopic): A persistent, scene-wide latent workspace ("canvas") acting as a cognitive map. It consists of a fixed grid of tokens ( $H \times W$ $H \times W$ ) tiling the scene coordinates $[-1, +1]^2$ $[- 1, + 1]^{2}$ , plus non-spatial register tokens.
- Key Innovation: The canvas tokens never undergo self-attention or Multi-Layer Perceptron (MLP) operations. They evolve solely through interaction with the backbone, ensuring low latency and scalability to large scenes.

Key Mechanisms

Canvas Attention: An asymmetric cross-attention mechanism that alternates between Read (backbone queries canvas) and Write (canvas queries backbone) operations.
- Asymmetry: Projections (Q, K, V, O) are applied only to the backbone side. The canvas side uses LayerNorm and RoPE but no learnable projections. This drastically reduces FLOPs, as the canvas token count is high ( $N_{can}$ ) while the backbone token count is low ( $N_{bb}$ ).
- Efficiency: This design allows the model to handle high-capacity working memory without the quadratic cost of self-attention on the canvas.
Scene-Relative Rotary Position Embeddings (SR-RoPE): Instead of standard positional embeddings, CanViT uses RoPE based on the scene's $[-1, +1]^2$ coordinate system. This binds the retinotopic backbone (view-dependent) and the spatiotopic canvas (view-independent) into a shared reference frame, allowing generalization across different zoom levels and viewpoints.
Viewpoint Encoding (VPE): A dedicated token encoding the current viewpoint $(x, y, s)$ using a scale-invariant parameterization $(x/s, y/s, \log s)$ , lifted via Random Fourier Features. This aids in policy learning and reconstruction.

Pretraining Strategy: Passive-to-Active Distillation

To train CanViT without labels or RL, the authors propose Policy-Agnostic Passive-to-Active Dense Latent Distillation:

Teacher: A frozen, high-resolution DINOv3 model acts as the teacher, providing ground-truth dense features (patch tokens and CLS token) for the entire scene.
Student: CanViT is trained to reconstruct these DINOv3 features from a sequence of low-resolution, random glimpses.
Objective: Minimize the reconstruction loss between the student's canvas tokens and the teacher's dense features in latent space (not pixel space).
Policy Agnosticism: The model is trained with randomized rollout policies (Random-IID and Full-IID) and truncated Backpropagation Through Time (BPTT) to ensure robustness to any viewing strategy.

3. Key Contributions

First AVFM: Introduction of CanViT, a general-purpose foundation model for active vision that is task-agnostic and policy-agnostic.
Novel Architecture: The Canvas Vision Transformer, featuring a persistent spatiotopic memory (canvas) that interacts with a retinotopic backbone via asymmetric cross-attention, eliminating canvas-side self-attention and MLPs for efficiency.
Training Paradigm: A label-free pretraining scheme using dense latent distillation from a passive teacher (DINOv3), bypassing the need for complex RL pipelines.
Efficiency: Achieves state-of-the-art performance with significantly fewer FLOPs compared to previous active models and competitive passive models.

4. Experimental Results

The model was pretrained on 13.2 million ImageNet-21k scenes (1 billion glimpses) in 166 hours on a single H100 GPU.

Semantic Segmentation (ADE20K)

Single Glimpse: A frozen CanViT-B achieves 38.5% mIoU with a single low-resolution glimpse.
- Outperforms the best previous active model (AME) which achieved 27.6% mIoU but required 19.5x more FLOPs.
- Surpasses its own DINOv3 teacher (33.2% mIoU) despite using a much smaller input resolution.
Multi-Glimpse: With a Coarse-to-Fine (C2F) policy and 21 glimpses, CanViT-B reaches 45.9% mIoU.
Zero-Shot Generalization: The model generalizes to unseen policies (e.g., Entropy-guided C2F), longer rollouts ( $T=21$ ), and higher resolutions (1024px) without fine-tuning.

Image Classification (ImageNet-1k)

Achieves 81.2% top-1 accuracy with the C2F policy and 21 glimpses (frozen weights, linear probe).
Outperforms other active models (AdaGlimpse: 77.5%) and approaches the performance of models trained with full RL and task-specific fine-tuning.

Efficiency

FLOPs: CanViT-B uses ~15.9 GFLOPs per glimpse, whereas previous active models (AME) require hundreds of GFLOPs for dense prediction.
Latency: Inference latency scales linearly with the number of glimpses and is significantly faster than passive models at high resolutions due to the fixed-size glimpse processing.

5. Significance

Bridging the Gap: CanViT successfully narrows the performance gap between passive and active vision, demonstrating that active vision can match or exceed passive models in accuracy while being far more computationally efficient.
New Research Axis: It establishes Active-Vision Foundation Models (AVFMs) as a viable research direction, proving that "how to see" can be learned independently of "where to look."
Practical Deployment: By removing the dependency on Reinforcement Learning for pretraining and using a memory-efficient architecture, CanViT provides a blueprint for real-world deployment in robotics and embodied AI, where sequential, resource-constrained perception is critical.
Scalability: The architecture supports arbitrary scene resolutions and rollout lengths, making it suitable for high-resolution video processing and dynamic environments.

In summary, CanViT represents a paradigm shift in active vision, moving from specialized, RL-dependent agents to general-purpose foundation models capable of building rich, persistent scene representations through efficient, sequential glimpses.