🤖 machine learning

Elastic Attention Cores for Scalable Vision Transformers

This paper introduces VECA (Visual Elastic Core Attention), a Vision Transformer architecture that achieves linear computational complexity and scalable high-resolution processing by replacing quadratic all-to-all self-attention with an efficient core-periphery structure where patch tokens communicate exclusively through a small, learned set of core embeddings.

Original authors: Alan Z. Song, Yinjie Chen, Mu Nan, Rui Zhang, Jiahang Cao, Weijian Mai, Muquan Yu, Hossein Adeli, Deva Ramanan, Michael J. Tarr, Andrew F. Luo

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alan Z. Song, Yinjie Chen, Mu Nan, Rui Zhang, Jiahang Cao, Weijian Mai, Muquan Yu, Hossein Adeli, Deva Ramanan, Michael J. Tarr, Andrew F. Luo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to organize a massive party with thousands of guests (the pixels in an image).

The Old Way: The "Everyone Talks to Everyone" Party

Traditional AI models, called Vision Transformers (ViTs), work like a chaotic party where every single guest must introduce themselves to every other guest to understand the room.

The Problem: If you have 100 guests, that's 10,000 handshakes. If you have 1,000 guests (a high-resolution photo), that's 1,000,000 handshakes. The time it takes to organize this grows explosively (quadratically). It's so slow and expensive that these models struggle with high-definition images.
The Assumption: The old models assumed that for the computer to "see" an object, every pixel needed to chat directly with every other pixel.

The New Way: VECA (The "VIP Core" Party)

The authors of this paper, Alan Song and colleagues, say: "Wait a minute. Do we really need everyone to talk to everyone?" They propose a new system called VECA (Visual Elastic Core Attention).

Think of VECA as a party with a VIP Core Group and a General Guest List.

The VIP Cores: Instead of thousands of guests talking to each other, the model creates a small, fixed group of "VIPs" (called Core Tokens). Let's say there are only 64 VIPs.
The Communication Rule:
- The Guests (image patches) only talk to the VIPs. They don't talk to each other directly.
- The VIPs talk to everyone (all the guests and other VIPs).
- The Result: Information flows from Guest → VIP → Guest. The guests never need to shake hands with each other.
The Efficiency Gain:
- In the old way, doubling the number of guests quadrupled the work.
- In VECA, doubling the guests only doubles the work (linear growth). It's like having a small, efficient team of managers (the VIPs) handle the chaos, rather than forcing every employee to manage every other employee.

The "Elastic" Superpower

The coolest part of VECA is that it's elastic (stretchy).

The Training: During training, the model learns to rank its VIPs. Some VIPs are "Super VIPs" who know the most important stuff (like "there is a dog here"), while others are "Junior VIPs" who know finer details (like "the dog has a floppy ear").
The Inference (The Party in Action):
- Need speed? You can tell the model, "Only use the top 8 VIPs." The model instantly runs faster because it's ignoring the junior staff. It might be slightly less detailed, but it's very quick.
- Need high quality? You can say, "Use all 64 VIPs." The model slows down a bit but gives you a super-detailed, high-precision answer.
- No Retraining: You don't need to build a new model for speed or quality. You just stretch or shrink the number of VIPs on the fly.

What Did They Find?

The authors tested this on a variety of tasks, like recognizing what's in a photo (classification) and drawing outlines around objects (segmentation).

Performance: Even with the "shortcut" of not letting pixels talk directly, VECA performed almost as well as the best, most expensive models currently available (like DINOv3).
The "Magic" Discovery: They noticed that the VIPs (Core Tokens) naturally learned to act like object detectors. Without being explicitly told to do so, the VIPs started grouping together to represent specific things, like "eggs in a bowl" or "a car wheel." They evolved from being vague blobs of information into specific, semantic groups.
Resolution: The model works great on small images and scales up to huge, high-resolution images without crashing or getting too slow, unlike the old models.

The Bottom Line

The paper claims that we don't need the expensive, quadratic "everyone talks to everyone" method to build smart vision AI. By using a small, smart team of "Core" tokens to mediate communication, we can build models that are:

Faster and cheaper (especially for high-resolution images).
Flexible (you can trade speed for accuracy on the fly).
Just as smart as the current state-of-the-art giants.

It's a new building block for the future of computer vision that makes high-resolution AI practical and efficient.

Technical Summary: Elastic Attention Cores for Scalable Vision Transformers

Problem Statement

Vision Transformers (ViTs) have achieved strong data-driven scaling by leveraging all-to-all self-attention, which allows for global receptive fields and flexible representation learning. However, this flexibility incurs a computational cost that scales quadratically ( $O(N^2)$ ) with image resolution, where $N$ is the number of image patches. This quadratic scaling becomes prohibitive in high-resolution domains, limiting the applicability of standard ViTs.

The prevailing assumption in ViT design is that dense, pairwise token interactions are necessary to learn rich visual-semantic representations. This paper challenges that assumption, positing that effective visual representations can be learned without direct patch-to-patch interactions.

Methodology: VECA (Visual Elastic Core Attention)

The authors propose VECA, a vision transformer architecture that replaces standard dense self-attention with a block-sparse core-periphery attention mechanism.

Core Architecture

Core-Periphery Structure: The architecture introduces a small, learned set of $C$ "core" tokens that act as a communication interface. The image patches form the "periphery."
Interaction Mechanism:
- Core Tokens: Form a fully connected clique, attending to all other core tokens and all patch tokens.
- Patch Tokens: Attend only to the active core tokens. They do not attend directly to other patch tokens.
- Information Flow: All information exchange between patches is routed exclusively through the core tokens.
Complexity: The attention cost is reduced from $O(N^2)$ to $O(2NC + C^2)$ . For a predetermined core count $C$ (where $C \ll N$ ), the complexity scales linearly ( $O(N)$ ) with image resolution.
Spatial Dynamics: Core tokens are initialized with learnable 2D coordinates and are updated across layers via a lightweight coordinate head. They utilize 2D axial Rotary Positional Encoding (RoPE), allowing them to maintain both semantic representations and evolving spatial positions.

Elastic Training and Inference

Nested Dropout: During training, the model employs nested dropout on the ordered bank of core tokens. The model is trained with a variable budget of active cores ( $C$ ), sampled from a distribution.
Implicit Ranking: This procedure implicitly ranks core tokens by importance. Early cores are active across all budgets and learn to encode the most essential, broadly useful visual information. Later cores specialize in finer spatial details or specific semantic concepts.
Inference Flexibility: At test time, the number of active cores can be adjusted without retraining. This allows for an elastic trade-off between computational cost (speed) and accuracy. Reducing $C$ yields coarser attention maps but significantly lower FLOPs.

Training Objective

The model is trained using feature distillation from a frozen DINOv3 teacher. The loss function aligns both global image representations (from the first core token) and dense patch representations with the teacher's targets, using unlabeled Objects365 images.

Key Contributions

VECA Architecture: A visual backbone that replaces quadratic self-attention with linear-time core-periphery attention in every layer, forcing token interactions through a resolution-invariant bottleneck.
Performance vs. Cost: Evaluation on classification and dense spatial tasks demonstrates that VECA achieves performance competitive with state-of-the-art foundation models (specifically DINOv3) while significantly reducing computational cost.
- Classification: At ViT-Base size, VECA achieves 81.93% Top-1 accuracy on ImageNet-1k, within ~1.6% of DINOv3 (83.56%), despite using only a fraction of the attention interactions.
- Dense Prediction: On the PASCAL Context segmentation benchmark, VECA achieves 57.46 mIoU compared to DINOv3's 57.74 mIoU, despite having 87.1% fewer attention interactions per layer.
Emergent Behaviors: The authors identify that core tokens develop object-centric representations and evolve from isotropic (spherical) distributions in early layers to semantically clustered structures in deeper layers, despite no explicit supervision for such behavior.

Results

Scalability: VECA maintains stable performance across varying input resolutions (256px to 1024px), whereas standard ViTs suffer from quadratic FLOP growth. VECA shows linear scaling in FLOPs and latency as resolution increases.
Elasticity: The model can elastically trade accuracy for speed. For instance, reducing the core budget from $C=64$ to $C=8$ retains significant performance (e.g., >96% of full model performance on classification) while drastically cutting compute.
Dense Features: Visualizations (UMAP) confirm that VECA produces stable, high-quality dense features that remain robust at higher resolutions, comparable to DINOv3.

Significance and Claims

The paper claims that direct patch-to-patch self-attention is not strictly necessary for learning high-quality dense visual representations. By routing information through a compact set of learned core tokens, VECA establishes elastic core-periphery attention as a scalable alternative building block for Vision Transformers.

The work suggests that the flexibility of all-to-all attention can be replaced by a structured, linear-time interface that preserves spatial alignment and dense feature quality. This offers a promising path for deploying Vision Transformers in high-resolution domains where computational efficiency is critical, enabling models to adapt their inference cost dynamically based on available resources or task requirements.