🤖 machine learning

Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models

This paper introduces PORTA, a retraining-free, task-agnostic pruning framework for vision-language models that leverages activation variation and adaptive sparsity allocation to achieve high compression rates while maintaining broad downstream performance without requiring task-specific data or fine-tuning.

Original authors: Minseok Kang, Hyunwoo Kim, Chanyoung Kim, Minwoo Kim, Jaekoo Lee, Dahuin Jung

Published 2026-08-10

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Minseok Kang, Hyunwoo Kim, Chanyoung Kim, Minwoo Kim, Jaekoo Lee, Dahuin Jung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot brain that can look at a picture and read a sentence, then instantly understand how they connect. This is the world of Vision-Language Models (VLMs). Think of these models as giant, hungry libraries that have read every book and looked at every photo on the internet. They are amazing at solving puzzles, answering questions, and finding specific images, but there's a catch: they are incredibly heavy. They require massive computers with huge amounts of memory and power to run, kind of like trying to carry a library in your backpack while hiking up a mountain. Because of this, it's very hard to put these smart robots on smaller devices like phones or tablets.

To fix this, scientists use a technique called pruning. Imagine the robot's brain is a massive web of connections. Pruning is like carefully snipping away the tiny, useless threads that don't help the robot think, leaving only the strong, important ones. This makes the robot lighter and faster without making it forget how to do its job. However, most existing ways of deciding which threads to cut are like using a map meant for a different country; they often need specific practice tests (data) for every single task the robot might face, or they get confused when the robot switches from looking at pictures to reading words. This paper tackles the problem of how to trim these giant brains once, perfectly, so they can run anywhere without needing to be retrained or re-tuned for every new job.

The "One-and-Done" Brain Trim

Meet PORTA (Prune-Once: Retraining-free Task-Agnostic pruning), a new method introduced by researchers Minseok Kang and his team. Their goal was to solve a frustrating problem: current pruning methods often act like a tailor who needs to measure you for a suit every time you want to wear a different shirt. If you want the robot to do image classification, you prune it one way; if you want it to do retrieval, you have to prune it again. This is slow and expensive. PORTA is different. It's like a master tailor who looks at the fabric once, cuts it perfectly, and then the resulting suit fits you for any occasion—whether you're running a race, sitting in a meeting, or dancing at a party.

The paper suggests that the main reason other methods fail at this "one-and-done" approach is that they rely on the size (magnitude) of the robot's internal signals to decide what to keep. The authors argue that in these giant models, the size of a signal can be misleading because the "language" part of the brain and the "vision" part of the brain behave differently. It's like trying to judge the importance of a shout in a library versus a shout in a stadium; the volume (magnitude) is different, but the meaning might be the same. Because of this, methods that just look at volume often cut the wrong things, especially when the robot is asked to do a task it hasn't seen before.

The Secret Sauce: Variance over Volume

Instead of listening to how loud a signal is, PORTA listens to how changeable it is. The researchers call this activation variation.

Here is the analogy: Imagine a group of students in a classroom.

The Old Way (Magnitude): The teacher asks, "Who is the loudest?" and keeps only the students who shout the most. But maybe the loudest student is just shouting because they are excited about one specific topic, while the quiet student is actually paying attention to everything the teacher says.
The PORTA Way (Variance): The teacher asks, "Who changes their answer the most depending on the question?" The student who adapts their answer based on the topic is the one who truly understands the material.

In the robot's brain, PORTA looks for features that show a lot of variation (fluctuation) across different inputs. If a part of the brain reacts differently to many different pictures or words, it's doing important work. If a part of the brain stays the same no matter what you show it, it's probably just dead weight. By using this "changeability" score, PORTA can figure out which parts of the robot are truly useful, regardless of whether it's looking at a cat or reading a poem. This makes the pruning "modality-agnostic," meaning it treats the vision and language parts fairly, without biasing the cut toward one side.

The Smart Cut: Not All Layers Are Equal

Once PORTA knows which features are important, it has to decide how much to cut from each layer of the robot's brain. The brain isn't a uniform block; some layers are like the foundation of a house, and others are like the roof. You wouldn't cut the foundation as aggressively as the roof.

PORTA uses a clever adaptive sparsity mechanism. It looks at how much "work" each layer is doing based on the variation of its output.

If a layer is doing a lot of heavy lifting (high variation in its output), PORTA says, "Don't touch this much!" and gives it a low pruning ratio (keeps most of it).
If a layer is just coasting (low variation), PORTA says, "We can cut this down significantly."

This ensures that the robot doesn't lose its ability to think just because we tried to make it smaller. The paper shows that this method prevents the "performance cliffs" where other methods suddenly crash when you try to cut too much.

The Results: Stronger, Lighter, Ready for Anything

The researchers tested PORTA on three famous robot brains: CLIP, BLIP, and Qwen2-VL. They didn't just test it on one task; they threw everything at it:

Image Classification: Identifying what's in a picture (like telling the difference between a flower and a dog).
Retrieval: Finding the right picture for a text description (or vice versa).
VQA (Visual Question Answering): Answering complex questions about an image.

The results were impressive. Even when they cut away 65% of the robot's parameters (making it very small), PORTA kept the robot performing much better than other methods.

On the CLIP model, at 65% sparsity, PORTA improved the average performance by 12.6% compared to a method called SparseGPT and by 21.5% compared to Wanda.
On BLIP (at 45% sparsity), PORTA achieved a mean score of 90.26, beating SparseGPT's 89.95.
On Qwen2-VL (at 50% sparsity), PORTA hit an average accuracy of 54.30, beating the next best method (Wanda) by 0.87 percentage points.

Perhaps most importantly, PORTA is robust. The paper shows that it doesn't matter if you use a small set of pictures to "calibrate" the pruning or a large set, or if you use pictures from one website or another. PORTA stays stable, whereas other methods swing wildly in performance depending on the data used.

The Bottom Line

PORTA suggests that we don't need to retrain these giant models to make them smaller. By simply looking at how much the robot's internal signals change rather than how loud they are, and by cutting more from the layers that do less work, we can create a single, slimmed-down model that is ready for any job. It's a "prune once, use forever" approach that could finally let these super-smart vision-language models run on the devices we carry in our pockets, without needing a supercomputer to power them. The authors confirm this through extensive zero-shot experiments, showing that a model pruned with PORTA can jump straight into new tasks without any extra training, maintaining high accuracy even when heavily compressed.

Technical Summary: PORTA – Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models

Problem Statement

Vision-Language Models (VLMs) have demonstrated remarkable generalization across multimodal tasks through large-scale pre-training. However, their rapidly increasing parameter counts and computational requirements hinder deployment in resource-constrained environments. While model compression techniques like pruning are essential, existing strategies face significant limitations in the context of VLMs:

Task-Specific Dependency: Most pruning methods rely on task-specific criteria or calibration data. When the downstream task changes, the model often requires re-pruning or re-training, introducing non-negligible overhead.
Modality Bias: Many approaches adopt importance measures derived from Large Language Models (LLMs), such as Wanda and SparseGPT, which rely on activation magnitude and outlier patterns. These patterns do not manifest identically in vision encoders, leading to modality-specific bias where pruning disproportionately affects either the vision or language branch.
Instability: LLM-oriented importance estimates are highly sensitive to the choice of calibration data. In task-agnostic settings where task-specific samples are unavailable, these methods yield unstable importance estimates and suboptimal pruning outcomes.

Consequently, there is a lack of a robust, task-agnostic pruning framework that can prune a VLM once using general data and deploy it across diverse downstream tasks without retraining or re-pruning.

Methodology: PORTA

The authors propose PORTA (Prune-Once: Retraining-free Task-Agnostic pruning), a framework designed to address modality bias and calibration sensitivity through two core components: a modality-agnostic importance formulation and an adaptive sparsity allocation mechanism.

1. Variance-Based Importance Formulation

Existing methods primarily use activation magnitude for importance estimation. The authors argue that magnitude is sensitive to modality-specific scale differences (e.g., text vs. vision activation distributions). Instead, PORTA utilizes activation variation (variance) as a more stable, modality-agnostic signal.

Rationale: Features with high variance respond broadly across diverse inputs, capturing generalizable structures, whereas low-variance features encode localized or input-specific patterns.
Formulation: For a layer with input activations $X \in \mathbb{R}^{B \times T \times D_{in}}$ and weights $W$ , the importance score $S_{i,j}$ for weight $W_{i,j}$ is calculated by combining the absolute weight magnitude with the variance of the corresponding input feature $j$ :
$v_j = \mathrm{Var}_{b,t}(X_{b,t,j})$
$S_{i,j} = v_j \cdot |W_{i,j}|$
This formulation ensures that features contributing significantly to the model's representational capacity across diverse inputs are retained, while mitigating bias caused by differing activation scales between vision and language branches.

2. Adaptive Sparsity Allocation

PORTA avoids uniform sparsity by assigning layer-wise pruning ratios based on the variability of the layer's output feature space.

Mechanism: The method estimates the variance of the output channels to determine the "utilization" of a layer's representational capacity. Layers with higher output variance (indicating higher utilization) are assigned lower sparsity ratios to preserve performance, while layers with lower variance are pruned more aggressively.
Efficiency: To ensure computational efficiency, the input covariance matrix is approximated as diagonal, allowing the reuse of variance statistics computed during the importance estimation stage.
Algorithm: The layer-wise scores are normalized and adjusted via a strength parameter $\alpha$ to meet a target global sparsity level (detailed in Algorithm 1 of the paper).

Key Contributions

A Retraining-Free Framework: PORTA introduces a "prune-once" paradigm where a pretrained VLM is pruned once using general calibration data and directly applied to diverse downstream tasks (classification, retrieval, VQA) without any additional training or re-pruning.
Modality-Agnostic Importance Criterion: The paper proposes a new importance metric based on activation variance rather than magnitude. This addresses the modality bias inherent in LLM-derived metrics, providing a stable signal across both vision and language encoders.
Adaptive Sparsity Allocation: A layer-wise pruning ratio mechanism is introduced that adapts to the specific representational utility of each layer, overcoming the limitations of uniform sparsity.
Robustness to Calibration Data: The framework demonstrates stability across different calibration datasets and sample sizes, making it practical for scenarios where task-specific data is unavailable.

Experimental Results

The authors evaluated PORTA on three representative VLM architectures: CLIP (OpenAI ViT-bigG), BLIP, and Qwen2-VL. Evaluations were conducted in a fully zero-shot setting across image classification, image-text retrieval, and Visual Question Answering (VQA).

Performance under High Sparsity: PORTA consistently outperformed baselines (Wanda, SparseGPT, ECoFLaP, Multiflow) at high sparsity levels (55%–65%).
- On CLIP at 65% sparsity, PORTA improved mean performance by 12.6% over SparseGPT and 21.5% over Wanda across eight metrics.
- On BLIP at 45% sparsity, PORTA achieved a higher mean retrieval score (90.26) compared to SparseGPT (89.95).
- On Qwen2-VL for VQA (ScienceQA) at 50% sparsity, PORTA achieved the highest average accuracy (54.30%), surpassing the second-best baseline (Wanda) by 0.87 percentage points.
Calibration Robustness: Experiments varying the calibration dataset (Flickr30K, MSCOCO, Visual Genome) and sample size showed that PORTA's performance remained stable (fluctuations <1%), whereas baselines exhibited significant performance shifts (up to ~5%).
Efficiency: PORTA completed pruning in approximately 195 seconds on CLIP, which is roughly 4x faster than SparseGPT and 2x faster than ECoFLaP, largely due to the reuse of variance statistics and diagonal covariance approximation.

Significance and Claims

The paper claims that PORTA advances the practical deployment of VLMs by enabling task-agnostic compression. By deriving importance from activation variation rather than magnitude, the method successfully mitigates the modality bias that plagues existing LLM-oriented pruning strategies.

The authors emphasize that PORTA allows a single pruned model to generalize across heterogeneous downstream applications (retrieval, classification, generation) without the need for re-pruning or retraining. This "prune-once" capability significantly reduces deployment overhead, particularly in scenarios where task-specific calibration data is difficult to obtain. The work validates that task- and modality-agnostic pruning is achievable with robust performance even under high compression levels, provided the importance estimation is grounded in modality-invariant representation utility.