💻 computer science

Understanding Task Transfer in Vision-Language Models

This paper introduces the Perfection Gap Factor (PGF) to systematically analyze task transferability in Vision-Language Models, revealing complex positive and negative transfer relationships across 13 perception tasks and providing actionable guidance for efficient training and data selection.

Original authors: Bhuvan Sachdeva, Karan Uppal, Abhinav Java, Vineeth N. Balasubramanian

Published 2026-04-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bhuvan Sachdeva, Karan Uppal, Abhinav Java, Vineeth N. Balasubramanian

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, multi-talented student named Vision-Language Model (VLM). This student is great at reading books, looking at pictures, and answering general questions. However, they aren't perfect at specific, tricky visual tasks like counting objects in a crowd, judging how far away things are, or spotting fake images.

To fix this, you decide to give them a crash course (called finetuning) on one specific subject, like "Object Counting." You expect them to get better at counting. But here's the surprise: after the crash course, the student might accidentally get worse at other things, like judging depth or recognizing art styles. It's like if you trained a chef to be a master pastry chef, and suddenly they forgot how to chop vegetables properly.

This paper asks a big question: If we train our AI on one visual skill, how does it change its ability to do all the other visual skills?

Here is the breakdown of their findings using some everyday analogies:

1. The "Perfect Score" Ruler (The PGF)

The researchers needed a way to measure these changes fairly. Imagine you have two students:

Student A is already a genius (98% correct). If you train them and they go to 99%, that's a huge deal!
Student B is a beginner (40% correct). If you train them and they go to 50%, that's a big jump in raw numbers, but they still have a long way to go.

The paper introduces a new ruler called the Perfection Gap Factor (PGF). Instead of just counting points, this ruler measures how much of the "remaining gap" to perfection was closed.

If you help the genius student get that last 1% closer to perfection, the PGF says, "Wow, that's a massive win!"
If you help the beginner student, the PGF says, "Good progress, but there's still a lot of room to grow."
This helps compare apples to oranges, ensuring we don't get fooled by raw numbers.

2. The "Social Network" of Skills

The researchers mapped out how these 13 different visual skills (like counting, depth, art style, etc.) talk to each other. They found that these skills aren't isolated islands; they are like a social network with distinct groups:

The "Donors" (The Generous Teachers): Some skills are like generous teachers. If you train the AI on Relative Depth (judging distance) or Visual Similarity, it tends to help the AI get better at many other skills too. These are "positive transfer" skills.
The "Pirates" (The Jealous Competitors): Some skills are like jealous pirates. If you train the AI on Functional Correspondence (matching parts of objects), it might actually steal the AI's ability to do other things, making it worse at counting or spotting fakes.
The "Sponges" (The Absorbers): Some skills are like sponges. If you train the AI on anything else, these skills (like Visual Similarity) soak up the benefits and get better automatically.
The "Sieves" (The Leaky Buckets): Some skills are like sieves. No matter what you train the AI on, these skills (like Forensic Detection) tend to leak out or get worse. They are very sensitive to change.

3. The "Clique" Effect

Just like in a high school cafeteria, the researchers found "cliques"—groups of skills that hang out together and help each other.

The Art & Puzzle Clique: Skills like Art Style, Jigsaw puzzles, and Visual Similarity form a tight-knit group. If you train the AI on one, the others get a boost. They share similar "mental muscles."
The Negative Clique: There are also groups of skills that fight each other. If you train on one, the others in the group suffer.

4. Bigger Brains, Better Friends

The study looked at three different sizes of AI models (Small, Medium, and Large).

The Finding: The bigger the model (the "brain"), the more positive transfer happens.
The Analogy: A small model is like a student with a small backpack; they can only carry one heavy book (skill) at a time, and adding another might drop the first one. A giant model has a massive backpack (more capacity). When you teach it one skill, it has enough room to organize that knowledge so it helps with many other skills at once.

5. Why This Matters (The "Recipe" for Training)

The most practical takeaway is about Data Selection.
Imagine you want to teach your AI to be a better "Object Counter," but you don't have enough counting data.

Old Way: Just guess and hope, or mix random data.
New Way (Using PGF): Look at the map! The researchers found that training on Relative Depth or Art Style actually helps "Object Counting" more than you'd think.
The Result: By picking the right "Donor" skills to train on, you can actually make the AI better at the target task than if you had trained it directly on the target task itself! It's like realizing that to get better at running, you should train on swimming first.

Summary

This paper is a map of the "mental geography" of AI vision. It tells us that training an AI isn't just about fixing one broken part; it's about understanding how that part connects to the whole body.

Don't just train blindly.
Use the "PGF" ruler to measure real progress.
Pick your "Donor" skills wisely to boost your AI's overall intelligence without accidentally breaking its other abilities.

In short: Teaching an AI one thing can make it a genius at many things, or a disaster at many things. This paper gives us the guidebook to make sure it becomes a genius.

1. Problem Statement

Vision-Language Models (VLMs) have achieved significant success in multimodal benchmarks (e.g., MMMU, DocVQA) but often lag behind humans and specialized models in fundamental visual perception tasks (e.g., depth estimation, object counting, visual similarity).

A critical challenge arises when practitioners attempt to improve VLMs on these specific tasks via fine-tuning. While fine-tuning on a source task ( $T_S$ ) improves performance on $T_S$ , it unpredictably affects the model's zero-shot performance on other perception tasks ( $T_j$ ).

The Gap: There is a lack of systematic understanding regarding how fine-tuning on one perception task reshapes the model's capabilities across a diverse suite of other perception tasks.
The Risk: Naive fine-tuning can lead to negative transfer, where improving one capability degrades others, potentially compromising the model's robustness in safety-critical applications.

2. Methodology

The authors propose a systematic framework to quantify and analyze task transferability in VLMs.

A. Experimental Setup

Models: Three open-weight VLMs from the Qwen-2.5-VL family (3B, 7B, and 32B parameters).
Tasks: 13 diverse perception tasks from the BLINK benchmark, categorized by:
- Perceptual Level: Low-level (e.g., Relative Depth), Mid-level, High-level.
- Granularity: Pixel-level, Crop-level, Image-level.
Protocol:
1. Fine-tune the base model on a single source task ( $T_S$ ) using LoRA (Low-Rank Adaptation).
2. Evaluate the fine-tuned model ( $M(T_S)$ ) in a zero-shot setting on all other target tasks ( $T_j$ ).
3. Repeat across multiple random seeds to ensure robustness.

B. The Core Metric: Perfection Gap Factor (PGF)

To compare transfer effects across tasks with vastly different baseline accuracies and difficulty ceilings, the authors introduce the Perfection Gap Factor (PGF).

Definition: PGF measures the fraction of the "remaining gap to perfection" that is closed (or opened) by fine-tuning.
$\mu_{i \to j} = \frac{Acc(M(T_i), T_j) - Acc(M, T_j)}{U_j - Acc(M, T_j) + \epsilon}$
Where:
- $Acc(M, T_j)$ : Baseline zero-shot accuracy.
- $Acc(M(T_i), T_j)$ : Accuracy after fine-tuning on source $T_i$ .
- $U_j$ : The upper-bound (ceiling) performance of task $T_j$ (set to 100% in experiments).
- $\epsilon$ : Small constant for numerical stability.
Advantages:
- Normalization: A +2% gain on a task near the ceiling (e.g., 90% $\to$ 92%) is weighted more heavily than the same gain on a low-performing task (e.g., 40% $\to$ 42%), reflecting the difficulty of closing the remaining gap.
- Boundedness: Positive PGF is capped at 1 (fully closing the gap), while negative PGF can be unbounded (depending on the number of evaluation questions), highlighting severe degradation.

C. Derived Metrics

Using PGF, the authors define two aggregate metrics:

Task Transferability ( $\Delta$ ): Measures how much a source task influences others. It captures both the breadth (number of tasks affected) and magnitude (average strength) of influence.
Task Malleability ( $\Theta$ ): Measures how susceptible a target task is to being influenced by other source tasks.

3. Key Contributions

Systematic Study of Zero-Shot Transfer: First comprehensive analysis of how single-task fine-tuning impacts zero-shot performance across a broad suite of perception tasks in modern VLMs.
Perfection Gap Factor (PGF): A novel, normalized metric that accounts for task difficulty and performance ceilings, enabling fair comparison of transfer effects.
Discovery of Task Personas and Cliques:
- Cliques: Identification of groups of tasks that mutually reinforce (Positive Cliques) or mutually degrade (Negative Cliques) each other.
- Personas: Classification of tasks into roles:
  - Donors: Tasks that consistently improve others (e.g., Semantic Correspondence).
  - Pirates: Tasks that consistently degrade others (e.g., Functional Correspondence).
  - Sponges: Tasks that readily absorb benefits from others (e.g., Visual Similarity, Relative Depth).
  - Sieves: Tasks that resist transfer or suffer degradation (e.g., Forensic Detection).
Data Selection Framework: Demonstrating that PGF can guide the selection of training data subsets to optimize performance on a target task without direct supervision, sometimes outperforming direct fine-tuning.

4. Key Results & Findings

A. Model Scale Effects

Positive Transfer Increases with Size: As model size increases (3B $\to$ 32B), the average positive transferability increases. Larger models possess greater representational capacity to capture generalizable features that benefit diverse tasks.
Negative Transfer: No consistent trend was observed for negative transferability across scales.

B. Task Characteristics

Low-Level Tasks are "Super-Donors": Fine-tuning on low-level tasks (Relative Depth, Relative Reflectance, Visual Correspondence) yields the highest positive transferability across other categories.
Image-Level Tasks are Highly Transferable: Tasks defined at the image level (e.g., Art Style, Counting) exhibit the highest positive transferability compared to pixel or crop-level tasks.
Malleability: Low-level tasks are the most "malleable" (benefit most from fine-tuning on other tasks).

C. Structural Patterns

Cliques: The authors identified stable Positive Cliques (e.g., Art Style, Counting, Jigsaw, Relative Depth, Visual Similarity) where tasks mutually boost each other. Conversely, Negative Cliques were found (e.g., Forensic Detection, Multi-view Reasoning, Object Localization, Semantic Correspondence), indicating inherent conflicts.
Video Generalization: These trends (Donors/Sponges) hold true when evaluating on spatio-temporal video tasks (VSI Bench), confirming the findings generalize beyond static images.

D. Practical Application: Data Selection

In scenarios where training data for a target task is unavailable, the authors used PGF to select a mixture of data from other related tasks.
Result: PGF-informed data selection consistently outperformed random sampling and, in cases like Jigsaw and Object Localization, even surpassed direct fine-tuning on the target task itself.

5. Significance and Implications

Guiding Efficient Training: The findings provide actionable guidelines for practitioners. Instead of blindly fine-tuning on a target task, one can leverage "Donor" tasks to improve a suite of capabilities simultaneously.
Mitigating Catastrophic Forgetting: By identifying "Pirate" tasks and "Negative Cliques," developers can avoid fine-tuning strategies that inadvertently degrade critical perception abilities (crucial for safety-critical domains like autonomous driving or medical imaging).
Understanding VLM Internals: The structured transfer graph suggests that VLMs do not learn tasks in isolation but through shared or competing representational substrates. Low-level features appear to be highly reusable building blocks for higher-level reasoning.
Future Directions: The paper highlights the need for benchmarks that measure inter-task correlations and suggests extending this analysis to open-ended generation and multi-lingual settings.

In summary, this paper moves beyond simple performance benchmarking to map the topology of task relationships in VLMs, offering a mathematical framework (PGF) and empirical evidence to optimize fine-tuning strategies for robust, general-purpose vision-language systems.