Technical Summary: DV-SFT (Direct Vision Supervision for Fine-Grained Visual Understanding)

Problem Statement

Multimodal Large Language Models (MLLMs) are typically trained end-to-end using a next-token prediction objective where supervision signals are applied exclusively to text tokens. Consequently, visual tokens—the primary carriers of visual information—are optimized only implicitly as part of the context. This lack of direct semantic guidance leads to coarse-grained visual understanding, where fine-grained details are often overlooked or hallucinated.

Existing attempts to improve visual precision by supervising visual tokens face significant limitations:

1. Indirectness: Methods relying on reward signals or representation alignment provide coarse or noisy supervision.
2. Objective Misalignment: Approaches like next-visual-token prediction or pixel-level reconstruction often misalign with the goal of visual understanding.
3. Architectural Complexity: Most methods require additional components (e.g., decoders, tokenizers) or extra forward passes, hindering practical applicability and maintaining a unified training paradigm.

There remains an open need for a direct, intuitive, and architecturally simple form of vision supervision that integrates seamlessly into standard MLLM training.

Methodology: Direct Vision Supervised Fine-Tuning (DV-SFT)

The authors propose DV-SFT, a method that constructs explicit, token-level supervision for visual tokens using the same next-token prediction objective used for text. The core hypothesis is that words describing the content of an image patch can serve as direct labels for the corresponding visual tokens.

Key Components

1. Vision Label Construction:
   To address the challenge of labeling visual tokens, the authors exploit the direct vision-text correspondence in OCR-related scenarios using two pipelines:

   • Image-to-Label: Uses text detection and recognition models to extract words and their spatial locations. Words are aligned to the visual token corresponding to their bottom-right corner.
   • Label-to-Image: Uses drawing tools to render documents from unimodal datasets onto a blank image patch grid, ensuring precise alignment between words and visual tokens.
   • Filtering: Words with excessive tokenization lengths or heights are discarded to ensure annotation precision.

2. Training Objective:
   DV-SFT treats the MLLM as a black box, requiring no architectural modifications. The total loss function combines standard text supervision ($L_t$) with direct vision supervision ($L_v$):
   $$L = L_t + \lambda L_v$$
   Where $L_v$ is the cross-entropy loss computed on visual tokens using their corresponding word labels. To ensure parallel training, only the first token of the vision label is used for loss computation.

3. Vision Smoothing:
   Since visual encoders possess bidirectional characteristics (unlike the causal nature of the LLM), a single visual token may contain semantic information from subsequent tokens. To address this, the authors introduce Vision Smoothing, inspired by label smoothing. This allows a visual token to learn its own label while also learning other vision labels present in the image, preventing information restriction.

Key Contributions

1. Novel Training Paradigm: The proposal of DV-SFT, which enables direct token-level supervision on visual tokens without modifying model architecture or forward paths, maintaining a fully unified, end-to-end training paradigm.
2. Label Construction Pipeline: The design of an offline pipeline leveraging OCR and drawing tools to generate fine-grained, token-level supervision for visual tokens.
3. Empirical Validation: Extensive experiments demonstrating that DV-SFT significantly improves visual capabilities and outperforms standard Supervised Fine-Tuning (SFT) and representation-based baselines (e.g., BASIC) across multiple benchmarks.

Experimental Results

The authors evaluated DV-SFT on three base models (Qwen3-VL-2B, Qwen3-VL-8B, and Qwen3-1.7B) across eight benchmarks.

• In-Domain Performance: DV-SFT achieved consistent advantages over standard SFT on document and chart understanding tasks (DocVQA, InfographicVQA, SQuAD). Notably, it outperformed the BASIC method, which suffered from model degradation on smaller models due to suboptimal representation quality.
• Out-of-Domain Generalization: DV-SFT demonstrated strong generalization on tasks such as ChartQA, OCRBench, OCRVQA, and TextVQA. The performance gains were more pronounced on out-of-domain tasks (average improvement of 1.6%) compared to in-domain tasks (0.74%), suggesting genuine enhancement of visual understanding rather than task-specific overfitting.
• Multimodal Alignment: Analysis of the training process showed that vision supervision accelerates the learning of visual features, particularly in the early stages of training and for unimodal models learning from scratch.
• Character Recognition: DV-SFT improved character recognition accuracy on both contextual and non-contextual (semantically meaningless) text recognition tasks, confirming the enhancement of fine-grained visual understanding.
• General Visual Capability: Evaluation on the MME benchmark (general visual perception and cognition) indicated that text-based vision supervision does not degrade general visual capabilities; rather, it slightly improves overall performance, suggesting the learned mechanisms generalize to non-text scenarios.

Significance and Claims

The paper claims that DV-SFT offers a practical and effective solution to the problem of coarse-grained visual understanding in MLLMs. By treating vision supervision as a standard next-token prediction task, the method:

• Eliminates Architectural Overhead: It requires no additional decoders, tokenizers, or forward passes, making it readily applicable to any existing MLLM pipeline.
• Enhances Fine-Grained Understanding: Direct supervision forces the model to align visual tokens with specific semantic content, reducing hallucinations and improving precision.
• Improves Alignment Efficiency: The method achieves higher multimodal alignment efficiency compared to indirect supervision methods.

The authors conclude that while their current implementation relies on OCR scenarios for label construction, the principle of direct vision supervision is a necessary step toward more robust MLLM training, with future work planned to extend these methods to broader visual domains beyond text.