Boosting Visual Instruction Tuning with Self-Supervised Guidance

1. Problem Statement

Multimodal Large Language Models (MLLMs), such as the LLaVA family, excel at general vision-language tasks but frequently fail at vision-centric problems requiring fine-grained visual reasoning (e.g., object counting, spatial positioning, geometric relations).

• Root Cause: The authors argue this limitation is not due to weak visual representations (modern encoders like CLIP and DINOv2 are powerful) but rather under-utilization of visual information during the instruction tuning phase.
• The "Language Shortcut" Phenomenon: Standard instruction tuning relies on image-instruction-response triplets generated from captions. Many of these tasks can be solved using language priors alone (e.g., guessing an answer based on common sense or the phrasing of the question) without actually analyzing the image. Consequently, models learn to rely on linguistic heuristics rather than visual evidence.
• Gap: Existing solutions often involve complex architectural changes, auxiliary losses, or Reinforcement Learning with Verifiable Rewards (RLVR), which are computationally expensive and difficult to integrate.

2. Methodology: V-GIFT

The authors propose V-GIFT (Visually Grounded Instruction Fine-Tuning), a lightweight framework that injects self-supervised learning (SSL) tasks directly into the visual instruction tuning dataset.

Core Concept

Instead of modifying the model architecture or loss functions, V-GIFT reformulates classical SSL pretext tasks as natural language instruction-following examples. These tasks are designed to be unsolvable by language priors alone, forcing the model to rely on visual tokens.

The Three SSL Tasks

The framework utilizes three specific tasks, converted into image-instruction-response triplets:

1. Rotation Prediction:
   • Task: The model is shown a rotated image and asked to identify the rotation angle (0°, 90°, 180°, 270°).
   • Visual Requirement: Requires recognizing object orientation and canonical poses; language priors cannot predict the specific rotation applied.
2. Point-wise Colorization (Color Matching):
   • Task: A color image is converted to grayscale with specific points labeled (A, B, C). The model must match these points to their original colors from a shuffled list of RGB candidates.
   • Visual Requirement: Requires fine-grained visual discrimination, spatial grounding, and reasoning about local appearance and global context.
3. Point Correspondence:
   • Task: Given two views of the same object, the model must identify which point in the second image corresponds to a query point in the first image.
   • Visual Requirement: Requires cross-view feature matching and spatial reasoning.

Integration Strategy

• Data Augmentation: The SSL tasks are generated automatically (no human annotation required) and mixed into the standard instruction tuning dataset ($\mathcal{D}_{inst}$).
• Training Ratio ($\rho$): A small fraction of the total data (typically 3% to 10%) consists of these SSL instructions.
• Optimization: The model is trained using standard autoregressive cross-entropy loss. No auxiliary losses, architectural changes, or separate pre/post-training stages are needed.

3. Key Contributions

1. Novel Framework: Introduced V-GIFT, which bridges the gap between self-supervised learning and instruction tuning by reformulating pretext tasks as natural language instructions.
2. Simplicity & Efficiency: The method requires no architectural modifications, no extra training stages, and no human annotations. It is plug-and-play for any MLLM pipeline.
3. Mitigation of Language Priors: By injecting tasks that must be solved visually, the method effectively reduces the model's reliance on language shortcuts.
4. Comprehensive Evaluation: Validated across multiple model backbones (Vicuna, Qwen), different model scales (LLaVA-1.5, LLaVA-OneVision-1.5), and various training regimes (Full Fine-tuning, LoRA).

4. Experimental Results

The authors evaluated V-GIFT on several vision-centric benchmarks: CVB-2D, POPE, MMStar, and BLINK.

• Performance Gains:
  • LLaVA-1.5 (Vicuna-7B): Improved average score by +0.9% (e.g., CVB-2D +2.6%).
  • LLaVA-1.5 (Qwen-7B): Improved average score by +0.9% (e.g., CVB-2D +1.1%).
  • LLaVA-OneVision-1.5: Improved average score by +1.2%, with a significant +3.4% gain on the BLINK benchmark.
  • LoRA Efficiency: Even with parameter-efficient fine-tuning (LoRA), V-GIFT achieved substantial gains (+2.2% average), outperforming complex methods like VIRAL which use auxiliary distillation losses.
• Ablation Studies:
  • Ratio Sensitivity: Performance peaks at low injection ratios (3–10%). Higher ratios yield diminishing returns.
  • Timing: Injecting SSL tasks during instruction tuning is crucial. Pre-training or post-training SSL stages resulted in no gain or catastrophic forgetting.
  • Compute Control: Control experiments confirmed that gains are not due to increased training iterations/compute, but specifically due to the nature of the visually grounded supervision.
  • Data Source: Gains were observed even when SSL tasks were generated from a single image (via augmentation), proving that dataset scale is less important than the presence of vision-forcing objectives.
• Qualitative Analysis:
  • TVI Scores: Models trained with V-GIFT showed higher "Token Visual Information" (TVI) scores, indicating reduced reliance on language priors.
  • Attention Maps: Attention visualizations showed V-GIFT models focusing more precisely on relevant objects (e.g., lamps, TVs) compared to baselines.

5. Significance and Conclusion

The paper demonstrates that the bottleneck in MLLM visual reasoning is often a training data distribution issue rather than a model capacity issue. By simply adjusting the instruction tuning distribution to include a small fraction of "vision-forcing" self-supervised tasks, models can be compelled to utilize visual evidence more effectively.

Significance:

• Paradigm Shift: Moves the focus from building more complex architectures to curating better supervision signals within existing pipelines.
• Accessibility: Provides a highly accessible method for improving MLLMs that does not require massive compute resources or complex RL pipelines.
• Future Direction: Suggests that low-level visual cues (geometry, color, correspondence) are essential for high-level multimodal reasoning and can be integrated via simple data augmentation strategies.

In summary, V-GIFT proves that "forcing" the model to look at the image through self-supervised tasks, even in a small proportion of the training data, significantly enhances its ability to perform fine-grained visual reasoning.