💻 computer science

Elastic ViTs from Pretrained Models without Retraining

This paper introduces SnapViT, a retraining-free, post-pretraining structured pruning method that leverages gradient information and an evolutionary algorithm to approximate cross-network correlations, enabling pretrained Vision Transformers to achieve elastic inference across a continuum of compute budgets without requiring labeled data.

Original authors: Walter Simoncini, Michael Dorkenwald, Tijmen Blankevoort, Cees G. M. Snoek, Yuki M. Asano

Published 2026-06-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Walter Simoncini, Michael Dorkenwald, Tijmen Blankevoort, Cees G. M. Snoek, Yuki M. Asano

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly smart library (a Vision Transformer or "ViT") that knows how to recognize almost anything in a picture. This library is so huge that it takes up a whole building and requires a giant, expensive generator to run it.

The problem? Most real-world devices (like your phone, a drone, or a smart camera) are small and have limited battery. They can't fit the whole library, and they can't afford the electricity to run it.

Usually, if you want a smaller library, you have to build a new, smaller one from scratch. But the authors of this paper, SnapViT, say: "Why build a new one? Let's just take the big library and instantly shrink it to fit any size you need, without losing its smarts."

Here is how they did it, explained simply:

1. The Problem: One Size Doesn't Fit All

Think of these AI models like a set of Russian nesting dolls (Matryoshka dolls). Usually, you only get a few specific sizes: Small, Medium, and Large. If your device needs something slightly smaller than "Medium" but bigger than "Small," you're out of luck. You either have to use the huge, slow one or a tiny, dumb one.

2. The Solution: "Snap" to Any Size

The authors created a method called SnapViT. It's like having a magical pair of scissors that can cut the big library down to any size you want (10% smaller, 50% smaller, etc.) in a single snap.

No Retraining: Usually, if you cut a piece off a machine, it stops working and needs to be fixed (retrained) for hours or days. SnapViT cuts the model and it works immediately. No fixing needed.
No Labels Needed: Most methods need a teacher to tell them what is important (like showing the AI thousands of pictures of cats and dogs). SnapViT is self-taught; it figures out what to keep just by looking at the patterns in the data itself.

3. How It Works: The "Smart Scissors"

To know which parts of the AI to cut and which to keep, the authors use a two-step scoring system:

Step 1: The Local Check (The "Hurt" Test)
Imagine poking the AI with a stick. If poking a specific part makes the AI stumble, that part is important. If it doesn't react, that part is probably useless. They use a "self-supervised" trick (looking at the same image from different angles) to see which parts of the brain are sensitive. This gives them a basic list of what to cut.
Step 2: The Global Check (The "Teamwork" Test)
This is the clever part. Sometimes, a part might look useless on its own, but it's actually a crucial teammate to another part. If you cut one, the other gets confused.
To find these hidden teams, they use a Genetic Algorithm (think of it as a digital evolution simulator). Instead of calculating every single connection (which would take forever), they simulate "what if we cut this?" and "what if we cut that?" thousands of times in a few minutes. They evolve a map of how different parts of the AI rely on each other.

4. The Result: Elastic Inference

Once they have this map, they can generate a "continuum" of models.

Need a model for a super-fast drone? SnapViT gives you a tiny version.
Need a model for a powerful server? SnapViT gives you a bigger version.
Need something in between? SnapViT gives you that too.

They tested this on several famous AI models (like DINO and SigLIPv2). They found that they could cut the model down by 40% (making it almost half the size) and it still worked almost as well as the original, with almost no loss in accuracy.

5. Why It's Fast

The authors claim they can do this entire process in less than five minutes on a single powerful computer chip (an A100 GPU). That's incredibly fast compared to other methods that might take days.

Summary Analogy

Imagine you have a giant, 100-page instruction manual for building a house.

Old way: If you only have time to read 50 pages, you have to rewrite the whole manual from scratch to make a 50-page version that still makes sense.
SnapViT way: You instantly highlight the 50 most important pages. You tear out the rest. The remaining 50 pages still tell you exactly how to build the house perfectly, and you did it in five minutes without needing a new architect.

This method allows anyone to take a massive, powerful AI and instantly shrink it to fit their specific needs, saving time, money, and energy, without needing to retrain the AI or have a teacher show it what to do.

Technical Summary: SnapViT – Elastic Vision Transformers from Pretrained Models without Retraining

Problem Statement

Vision foundation models have achieved state-of-the-art performance across diverse tasks but are typically released in a limited set of pre-determined sizes (e.g., specific parameter counts for DINOv3 or DeIT families). This rigidity forces practitioners to make sub-optimal deployment choices when facing real-world constraints on compute, latency, and cost. Existing solutions for model flexibility face significant limitations:

Knowledge Distillation: Requires a predetermined target architecture and often relies on non-public pretraining datasets, undermining robustness.
Elastic Inference: Methods like Matryoshka representations or Slimmable Networks require models to be designed with specific nested structures during pretraining, making them inapplicable to existing, off-the-shelf pretrained models.
Structured Pruning: While effective at reducing memory and compute, most current pruning techniques are tailored to specific tasks or constraints and typically require retraining, leaving a gap for universally adaptable models that do not need labeled data or a classification head.

Methodology: SnapViT

The authors propose SnapViT (Single-shot network approximation for pruned Vision Transformers), a post-pretraining structured pruning method that generates a continuum of elastic sub-networks from a single pretrained model without retraining or labeled data. The core of the method is a prunability score ( $P$ ) that ranks parameters for removal based on two components:

1. Local Hessian Approximation (Self-Supervised Gradients)

To capture local parameter sensitivity without a classification head, the method approximates the diagonal of the Hessian matrix using self-supervised gradients. Specifically, it employs the DINO objective to compute gradients over a dataset of images. The local curvature $H^{(l)}$ is estimated as the average squared norm of the gradients:
$H^{(l)} \approx \frac{1}{N_D} \sum_{i=1}^{N_D} \|\nabla_\theta L_i\|^2$
This provides a baseline measure of how critical individual parameters are within their specific blocks (e.g., feed-forward neurons or attention heads).

2. Global Hessian Estimation (Evolutionary Algorithm)

To address the limitation of ignoring inter-block dependencies (off-diagonal Hessian elements), SnapViT introduces a novel approximation using the Exponential Natural Evolution Strategy (xNES).

Instead of explicitly computing the intractable full Hessian, xNES learns a set of structure-wise scaling factors ( $c$ ) that reweight the local sensitivities.
The algorithm optimizes a multivariate Gaussian distribution over these scaling factors. The fitness function is label-free, measuring the cosine similarity between embeddings of the original model and the pruned model (after PCA compression).
Through natural-gradient updates, the covariance matrix ( $\Sigma$ ) of the evolutionary strategy evolves to approximate the inverse of the true inter-block Hessian ( $H^{(g)} \approx \alpha \Sigma^{-1}$ ). This allows the method to identify which blocks can be pruned jointly without degrading representation quality.

3. Unified Prunability Score and Elastic Generation

The final prunability score is the element-wise product of the local sensitivity and the global scaling factors:
$P = \text{diag}(H^{(l)}) \odot (M c)$
where $M$ is a membership matrix expanding block-level factors to individual parameters.

Single-Shot Generation: Once $P$ is computed, all parameters are globally ranked. This single ranking allows the extraction of sub-networks for any desired sparsity level ( $S \in [0, 1]$ ) without further optimization.
Scope: The method prunes both row-column combinations within feed-forward blocks and entire attention heads.

Key Contributions

Elastic Pruning Strategy: An efficient, fast pruning strategy for pretrained Vision Transformers that yields elastic models adaptable to any computational constraint without retraining.
Evolutionary Hessian Approximation: A novel strategy using a genetic algorithm (xNES) to approximate the off-diagonal components of the Hessian for network structures, capturing cross-block correlations that traditional diagonal approximations miss.
Label-Free, Head-Agnostic Performance: The method achieves state-of-the-art performance under significant pruning without requiring labeled data or a classification head, generalizing across supervised and self-supervised foundation models.

Experimental Results

The authors evaluated SnapViT on eight datasets (including ImageNet-1k, Pascal VOC, and various fine-grained benchmarks) using k-nearest neighbor (k-NN) classification, linear probing, and linear semantic segmentation.

Performance vs. SOTA: SnapViT consistently matches or outperforms state-of-the-art methods (LAMP, LLM Surgeon, FPTP, SparseGPT, SNIP Magnitude, NViT) across various sparsity levels.
- For DINOv1 ViT-B/16, the method prunes to 40% sparsity with less than 5% accuracy degradation, achieving a 1.58x inference speedup.
- At 50% sparsity on supervised AugReg models, it outperforms SNIP by 7% and FPTP by 12.3% in linear classification.
- For SigLIPv2 and DINOv3 (large-scale foundation models), the method significantly outperforms baselines, with improvements of up to 34.3% in linear classification over second-best methods.
Efficiency: The entire pruning process for a model family (e.g., DeIT-III) takes less than five minutes on a single A100 GPU.
Post-Processing: The method is compatible with post-pruning weight correction (SparseGPT-style) and full fine-tuning. With a single weight correction step, performance at extreme sparsity (e.g., 50% on SigLIPv2) is largely preserved.
Scalability: While large-scale pretraining (e.g., DINOv3, SigLIPv2) makes pruning more challenging due to distributed representational knowledge, SnapViT remains effective, especially when combined with weight correction or longer optimization horizons (500 xNES iterations).

Significance and Claims

The paper claims that SnapViT bridges the gap between the rigidity of current foundation model deployments and the need for flexible, resource-adaptive inference. By enabling the extraction of a continuum of sub-networks from a single pretrained model in a single shot, it allows users to precisely tailor state-of-the-art models to their specific computational budgets.

The authors emphasize that their approach:

Eliminates the need for retraining, saving significant computational resources and time.
Removes dependency on labeled data, making it applicable to self-supervised foundation models that lack classification heads.
Captures global structural correlations via evolutionary approximation, overcoming the limitations of local-only pruning metrics.
Reduces environmental impact by lowering the compute requirements and power usage of vision transformers, potentially reducing CO2 emissions from inference.

The work is presented as a practical solution for deploying large-scale vision models in constrained environments while maintaining high performance across diverse downstream tasks.