🤖 machine learning

Accelerating Vision Transformers with Adaptive Patch Sizes

This paper introduces Adaptive Patch Transformers (APT), a method that dynamically adjusts patch sizes based on image content to significantly reduce token counts and accelerate ViT training and inference by up to 50% while maintaining performance across various high-resolution visual tasks.

Original authors: Rohan Choudhury, JungEun Kim, Jinhyung Park, Eunho Yang, László A. Jeni, Kris M. Kitani

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rohan Choudhury, JungEun Kim, Jinhyung Park, Eunho Yang, László A. Jeni, Kris M. Kitani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef preparing a massive, high-resolution photo of a city for a food critic. The critic (the AI model) needs to "taste" every part of the image to understand what it is.

The Old Way (Standard Vision Transformers):
Traditionally, the chef chops the entire photo into thousands of tiny, identical square tiles (like a mosaic), no matter what's in them.

The Problem: If the photo shows a clear blue sky, the chef still chops it into tiny tiles. If the photo shows a busy street with a crowd, the chef chops that into tiny tiles too.
The Result: The chef spends the same amount of time chopping the boring sky as the exciting street. The critic has to taste thousands of tiny, boring sky-tiles before getting to the interesting parts. It's slow, wasteful, and the kitchen gets crowded with too many small pieces.

The New Way (Adaptive Patch Transformers - APT):
This paper introduces a smarter chef named APT. Instead of using a fixed knife size, APT looks at the photo and decides how big to cut the tiles based on what's actually there.

The Core Idea: "Cut Big Where It's Boring, Cut Small Where It's Busy"

Think of the image as a landscape:

The "Sky" (Homogeneous Areas): If you see a large patch of blue sky or a blank white wall, there isn't much going on. APT says, "This is boring; I don't need to look at every inch." So, it cuts huge tiles (like 64x64 pixels) to cover that whole area with just one piece.
The "Crowd" (Complex Areas): If you see a person's face, a car, or a tree with leaves, there is a lot of detail. APT says, "This is important! I need to see the details." So, it cuts tiny tiles (like 16x16 pixels) to capture every nuance.

How It Works (The Magic Tricks)

1. The "Entropy" Smell Test
How does APT know what's boring and what's busy? It uses a math concept called entropy, which is basically a measure of "surprise" or "disorder."

Low Entropy (Boring): A blue sky has very low entropy. The pixels are all similar. APT smells this and says, "I can cover this with a big tile."
High Entropy (Busy): A face has high entropy. The pixels change color and shape rapidly. APT smells this and says, "I need small tiles to catch all this detail."

2. The "Zero-Weight" Safety Net
When you change how you cut the photo, the AI might get confused at first. To fix this, APT uses a clever trick called a Zero-Initialized MLP.

Analogy: Imagine you are teaching a student a new way to solve a math problem. You don't want them to forget the old way immediately. So, you give them a "safety net" that starts with zero weight. At first, the safety net does nothing, and the student relies on their old knowledge. As they practice (training), the safety net slowly learns to help them with the new method. This allows the AI to switch to the new, faster cutting style almost instantly without losing its intelligence.

3. The "Packing" Trick
Since some images now have huge tiles and some have tiny ones, the total number of pieces (tokens) varies. APT uses a technique called sequence packing.

Analogy: Instead of having 10 different boxes of different sizes that don't fit on the conveyor belt, APT packs all the pieces from one image into one long, continuous train. The AI processes this train efficiently, ignoring the empty spaces between different images. This keeps the factory running at full speed.

Why Should You Care? (The Results)

The paper shows that this method is a game-changer:

Speed: It makes the AI 40% to 50% faster. If a standard AI takes 10 seconds to look at a high-resolution photo, APT might do it in 5 or 6 seconds.
No Loss of Smarts: Even though it looks at fewer pieces, it doesn't miss anything important. It still gets the same accuracy on tests like recognizing objects, reading text in images, or finding cars in a video.
Easy to Install: You don't need to rebuild the whole AI from scratch. You can take an existing, trained AI and just "plug in" APT. It learns the new way in just one day (or even one training cycle).

Summary Metaphor

Imagine reading a book.

Standard AI: Reads every single letter, even the spaces between words and the blank pages at the end of chapters. It's thorough but incredibly slow.
APT: Skims the blank pages and the spaces between words (reading them as one big chunk), but slows down to read every single letter when it gets to the exciting plot twists or complex dialogue.

The Result: You get the whole story just as well, but you finish the book in half the time.

This paper proves that by being "smart" about how we slice up images, we can make AI faster, cheaper to run, and ready for the high-resolution world of tomorrow.

1. Problem Statement

Vision Transformers (ViTs) have become the dominant paradigm for visual recognition, but they suffer from scalability issues due to the quadratic computational cost of self-attention relative to sequence length.

Fixed-Size Limitation: Standard ViTs partition images into uniformly sized patches (e.g., $16 \times 16$ ), regardless of image content. This results in excessively long token sequences for high-resolution images, even in regions with high redundancy (e.g., a clear blue sky).
Inefficiency of Prior Solutions: Existing methods to reduce tokens often face two main drawbacks:
1. Fixed Reduction Ratios: Merging a fixed proportion of tokens (e.g., 50%) is suboptimal because it treats simple and complex regions equally.
2. Forward Pass Pruning: Methods that prune tokens dynamically during the forward pass introduce irregular shapes and padding, which often negate theoretical speedups in practice due to hardware inefficiencies.
The Gap: Unlike language models that use adaptive tokenizers (e.g., Byte-Pair Encoding) to handle variable subword frequencies, ViTs lack a mechanism to adaptively adjust token granularity based on visual complexity.

2. Methodology: Adaptive Patch Transformer (APT)

The authors propose APT, a content-aware framework that varies patch sizes within a single image to reduce the total number of input tokens while preserving critical information.

A. Adaptive Patch Sizing (Content-Aware Partitioning)

Instead of a fixed grid, APT uses a quadtree-like hierarchical structure to determine patch sizes based on local entropy.

Entropy Calculation: The algorithm computes the entropy of pixel intensity distributions at multiple scales ( $2^i \times p$ ). Low entropy indicates high redundancy (smooth areas), while high entropy indicates complexity (edges, textures).
Hierarchical Assignment:
1. Start with the largest possible patch size ( $2^S \times p$ ).
2. If a patch's entropy is below a tunable threshold $\tau$ , it is retained as a large patch.
3. If entropy is high, the patch is subdivided into smaller patches, and the process repeats until the base patch size ( $p \times p$ ) is reached.
Result: Homogeneous regions (e.g., backgrounds) are represented by large patches (fewer tokens), while complex regions (e.g., faces, objects) are represented by small patches (more tokens).

B. Patch Aggregation & Embedding

To feed variable-sized patches into a standard Transformer, APT must project them into a uniform embedding dimension ( $d_{embed}$ ).

Hybrid Embedding Strategy:
- Small Patches: Processed directly by the standard linear patch embedding layer.
- Large Patches: Processed via a dual-path approach:
  1. Sub-patch Aggregation: The large patch is split into its constituent base-sized sub-patches, embedded individually, and aggregated back to the base size using a convolutional layer.
  2. Resized Embedding: The original large patch is resized to the base size and embedded.
- Zero-Initialized MLP: The final embedding is the sum of the aggregated sub-patch features and the resized embedding, passed through a Zero-Initialized Multi-Layer Perceptron (ZeroMLP).
Significance of ZeroMLP: Inspired by ControlNet, initializing this connection with zeros allows the model to start with the behavior of the original resized ViT (preserving pre-trained weights) and gradually learn to incorporate high-resolution details during fine-tuning. This enables convergence in as little as 1 epoch.

C. Handling Variable Sequence Lengths

Since APT produces a variable number of tokens per image, it employs Sequence Packing:

Tokens from a batch of images are concatenated into a single long sequence.
A block-diagonal attention mask ensures tokens only attend to other tokens within the same image.
This approach is natively supported by efficient attention backends like FlashAttention, avoiding the overhead of irregular padding.

D. Adaptation to Dense Tasks

For tasks requiring dense predictions (Object Detection, Semantic Segmentation):

Feature Map Reconstruction: Larger patch tokens are repeated ( $2^{2i}$ times) to reconstruct a feature map with the correct spatial resolution, allowing the use of standard transposed convolutions for upsampling.
Window Attention: APT is compatible with window-based attention mechanisms (e.g., in Swin Transformers or ViTDet) by applying the adaptive patching strategy within fixed window boundaries.

3. Key Contributions

Content-Aware Tokenization: Introduced a method to dynamically assign patch sizes based on local entropy, significantly reducing input sequence length without manual tuning of reduction ratios.
Efficient Integration: Developed a zero-initialized embedding mechanism that allows APT to be applied to pre-trained ViTs with minimal fine-tuning (1 epoch) and negligible accuracy drop.
Broad Applicability: Demonstrated that APT works effectively across diverse tasks, including image classification, Visual Question Answering (VQA), object detection, and semantic segmentation.
Hardware Efficiency: By avoiding irregular token pruning and utilizing sequence packing with FlashAttention, APT achieves real-world speedups that theoretical FLOP reductions alone cannot guarantee.

4. Experimental Results

The paper reports significant speedups while maintaining or improving accuracy across various model scales (ViT-B, L, H) and resolutions.

Image Classification (ImageNet):
- Throughput: Achieved a 40% speedup on ViT-L and 50% on ViT-H during inference.
- Training Time: Reduced training time by up to 30% on high-resolution tasks.
- Accuracy: Matched the baseline performance of standard ViTs. When applied to an already fine-tuned checkpoint, APT recovered full performance after just 1 epoch of fine-tuning.
Downstream Tasks:
- Visual QA (LLaVA): Increased throughput by 22-23% while matching or exceeding baseline accuracy on benchmarks like GQA and VQAv2.
- Object Detection (COCO): Achieved a 30% speedup in training/inference on 1536x1536 resolution images with no loss in mAP.
- Semantic Segmentation (ADE20K): Reduced input tokens by 28-32%, accelerating inference by 11% while maintaining pixel-level fidelity.
Comparison: APT consistently outperformed layer-level token merging baselines (e.g., ToMe, EViT) and input-level baselines (random masking, simple resizing), particularly at higher resolutions and model scales.

5. Significance and Conclusion

Paradigm Shift: APT bridges the gap between NLP's adaptive tokenization and computer vision, proving that variable-granularity tokenization is superior to fixed-size splitting for visual data.
Practical Impact: It offers a "drop-in" solution for accelerating existing ViT models. The ability to converge in a single epoch makes it highly practical for practitioners who wish to reduce compute budgets without retraining models from scratch.
Scalability: The benefits of APT scale with image resolution and model size, making it particularly valuable for the emerging trend of high-resolution, large-scale vision models.
Limitations: The method currently relies on hand-crafted entropy thresholds (though they are robust) and does not yet support image generation tasks, which is identified as a direction for future work.

In summary, APT provides a robust, content-aware mechanism to accelerate Vision Transformers by intelligently reducing redundancy in homogeneous image regions, achieving substantial speedups in both training and inference without compromising model performance.