VisNec: Measuring and Leveraging Visual Necessity for Multimodal Instruction Tuning

This paper introduces VisNec, a principled framework that measures visual necessity to filter out redundant and misaligned samples from multimodal instruction datasets, enabling models to achieve superior performance with significantly less training data.

The Gist

Imagine you are trying to teach a very smart robot how to understand the world. You have a massive library of books and pictures, but here's the catch: not all of them are actually helpful.

Some books have pictures that are just decorations (you could guess the answer just by reading the text). Some books have pictures that actually contradict the text (the text says "sunny day," but the picture shows a storm). And some books are the real deal, where the picture is absolutely essential to solving the puzzle.

The paper "VisNec" is about a new, super-smart librarian who can sort through this massive library and pick out only the best, most necessary books to teach the robot.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Cheat Sheet" Effect

Imagine you are taking a test.

• The Redundant Question: "What color is grass?"
  • The Trap: You don't need to look at the picture of the grass to know the answer is "green." You just know it from your general knowledge. If the robot learns from this, it gets lazy. It stops looking at the pictures and starts just guessing based on the words.
• The Misaligned Question: "Is this a sunny day?" (But the picture shows a dark, rainy cave).
  • The Trap: The text says "Yes," but the picture says "No." If the robot tries to learn from this, it gets confused and starts hallucinating (making things up).

Current methods often just grab a random handful of questions from the library. This means the robot wastes time studying "cheat sheets" (redundant data) and gets confused by "bad instructions" (misaligned data).

2. The Solution: The "Blindfold Test" (VisNec)

The authors created a tool called VisNec (Visual Necessity Score). Think of it as a Blindfold Test for every single question in the library.

Here is the process:

1. The "Blind" Run: The robot tries to answer a question without looking at the picture (it's blindfolded). It records how hard it was to guess.
2. The "Sighted" Run: The robot tries to answer the same question with the picture. It records how hard it was this time.
3. The Score: VisNec calculates the difference.
   • High Score (Vision-Critical): The robot struggled when blindfolded but got it right with the picture. Verdict: "This picture is essential! Keep this sample."
   • Zero Score (Redundant): The robot got it right even when blindfolded. Verdict: "The picture is useless here. Discard this sample."
   • Negative Score (Misaligned): The robot did worse with the picture than without it. Verdict: "The picture is confusing or wrong. Throw this away immediately!"

3. The Strategy: The "Fair Menu" (Semantic Clustering)

If you just picked the top 15% of "High Score" questions, you might end up with a library full of only "geometry" questions and no "cooking" questions. The robot would become a geometry genius but a terrible cook.

To fix this, VisNec uses a Menu Strategy:

• It groups questions by topic (e.g., "Cooking," "Cars," "Animals").
• Inside each group, it picks the best "High Score" questions.
• Result: The robot gets a balanced diet of knowledge, but every single bite is packed with visual value.

4. The Result: Less is More

The paper tested this on huge datasets (like the LLaVA-665K, which has 665,000 samples).

• Old Way: Train on all 665,000 samples. Expensive, slow, and the robot learns some bad habits.
• VisNec Way: Train on only 15% of the samples (about 98,000), but they are the perfect 15%.

The Outcome:
The robot trained on the tiny, curated 15% subset actually performed better than the robot trained on the full dataset!

• It learned faster.
• It made fewer mistakes.
• It didn't get confused by bad data.

The Big Takeaway

VisNec proves that in the world of AI, quality beats quantity. Instead of drowning the robot in millions of mediocre examples, we should give it a smaller, curated set of examples where the pictures truly matter. It's the difference between feeding a student a whole encyclopedia they can't read versus giving them a few perfect, illustrated stories that teach them exactly what they need to know.

Technical Summary

1. Problem Statement

Multimodal Large Language Models (MLLMs) rely heavily on instruction tuning to learn cross-modal reasoning. However, existing large-scale instruction datasets (e.g., LLaVA-665K) suffer from two critical issues that hinder efficient and robust training:

• Visual Redundancy: A significant portion of samples can be answered using linguistic priors alone (text-only shortcuts) without needing visual input. Training on these wastes resources and encourages models to ignore visual evidence.
• Multimodal Misalignment: Some samples contain annotation errors or noisy image-text pairs where the visual content contradicts the text. Training on these actively degrades model performance and amplifies hallucinations.
• Inefficiency of Existing Selection Methods: Current data selection strategies (e.g., gradient influence, clustering, or text-only difficulty scores) often treat multimodal data holistically. They fail to explicitly distinguish the independent contribution of the visual modality, leading to the selection of "pseudo-multimodal" samples that reinforce linguistic shortcuts rather than true cross-modal reasoning.

2. Methodology: VisNec Framework

The authors propose VisNec (Visual Necessity Score), a principled data selection framework that quantifies the marginal contribution of visual input to a model's predictive uncertainty. The framework operates in two main stages:

A. Visual Necessity Score (VisNec) Calculation

Inspired by the theory of V-usable information, VisNec measures how much a visual input reduces predictive uncertainty beyond what is already available from the text.

1. Blind Forward Pass: For a given sample $(v, t, y)$ (image, text instruction, target response), the model performs a forward pass where the image tokens are replaced with padding tokens and their attention contributions are suppressed. This yields a Text-Only Loss ($\mathcal{L}_{\text{Blind}}$), representing the model's uncertainty using only linguistic context.
2. Multimodal Forward Pass: The model performs a standard forward pass with both image and text, yielding a Multimodal Loss ($\mathcal{L}_{\text{MM}}$).
3. Score Computation: The VisNec score ($S_{\text{VisNec}}$) is the difference between these losses:
   $$S_{\text{VisNec}} = \mathcal{L}_{\text{Blind}} - \mathcal{L}_{\text{MM}}$$
   • $S_{\text{VisNec}} > 0$ (Vision-Critical): The image significantly reduces error. The sample requires genuine visual reasoning.
   • $S_{\text{VisNec}} \approx 0$ (Redundant): The image adds no value; the answer is solvable via text alone.
   • $S_{\text{VisNec}} < 0$ (Misaligned): The image increases error (e.g., due to conflicting annotations), indicating noise or misalignment.

B. Semantic-Aware Stratified Sampling

To prevent the selected subset from biasing toward specific task types (e.g., selecting only high-visual-complexity geometric tasks while ignoring OCR), the framework employs a coarse-to-fine selection strategy:

1. Instruction Clustering: Textual instructions are embedded and clustered using K-Means ($K=20$) based on semantic intent (e.g., grouping "describe the image" vs. "solve the math problem").
2. Intra-Cluster Selection: Within each semantic cluster, samples with $S_{\text{VisNec}} \leq 0$ are discarded. The remaining samples are ranked by their VisNec scores, and the top $r\%$ (e.g., 15%) are selected. This ensures the final subset is both visually indispensable and task-diverse.

3. Key Contributions

1. Identification of a Critical Gap: The paper highlights that existing data selection methods neglect the independent contribution of the visual modality, leading to the retention of "pseudo-multimodal" samples that hinder true cross-modal reasoning.
2. VisNec Framework: Introduction of a lightweight, model-relative data selection method that explicitly quantifies the marginal value of visual input through counterfactual loss comparison (Blind vs. Multimodal).
3. State-of-the-Art Performance: Demonstration that selecting a small subset of data based on visual necessity achieves or surpasses full-data training performance across diverse benchmarks and model scales.

4. Experimental Results

The authors evaluated VisNec on LLaVA-665K and Vision-Flan-186K datasets, fine-tuning LLaVA-v1.5-7B and Qwen2.5-VL models across 10 benchmarks (including VQAv2, MME, MM-Bench, and POPE).

• Efficiency & Performance:
  • On LLaVA-665K, training on only 15% of the data selected by VisNec achieved 100.2% of the full-data performance, outperforming all baselines (Random, IFD, XMAS, CoIDO, etc.).
  • On Vision-Flan-186K, VisNec achieved 115.8% relative performance, significantly outperforming the full-data baseline.
• Generalization: The method generalized effectively across different model scales (3B, 7B, 32B) and architectures (LLaVA, Qwen2.5-VL), proving it captures intrinsic data value rather than model-specific biases.
• Ablation Studies:
  • Loss Components: Using only text-only or multimodal-only loss resulted in lower performance (95.6% and 94.0% respectively) compared to the full VisNec differential (100.2%), confirming the necessity of the contrastive signal.
  • Clustering: Removing semantic clustering (Top-VisNec only) dropped performance by ~3.2%, highlighting the importance of maintaining task diversity.
• Cost Analysis: VisNec is highly efficient. The total cost (selection + fine-tuning) was 23.0 GPU-hours, compared to 76.0 GPU-hours for full-data fine-tuning. It is significantly faster than other selection methods like Self-Filter (84.5 hours) and does not require expensive external LLM APIs.

5. Significance

VisNec represents a paradigm shift in multimodal data curation. Instead of simply filtering for "difficulty" or "diversity," it filters for visual necessity.

• Data Efficiency: It proves that "less is more" in multimodal tuning; a small, high-quality subset of truly visual-dependent samples is more effective than massive, noisy datasets.
• Robustness: By explicitly filtering out misaligned and redundant samples, VisNec improves model robustness and reduces hallucinations.
• Scalability: The method is computationally lightweight and architecture-agnostic, making it a practical solution for scaling next-generation MLLMs with limited computational resources.