Generalize LMMs to Versatile Visual Modalities via Fabricated Modality Synthesis

Technical Summary: Generalize LMMs to Versatile Visual Modalities via Fabricated Modality Synthesis

Problem Statement
Current Large Multimodal Models (LMMs) demonstrate strong performance on RGB vision but struggle to generalize to unseen visual modalities (e.g., thermal, depth, X-ray). Existing approaches typically rely on scaling data to include specific non-RGB modalities during training, creating a "generalization gap" where models fail on modalities not present in their training distribution. The core challenge is that LMMs often conflate invariant scene semantics with modality-specific visual appearances, forcing the model to learn new alignments for every distinct modality rather than developing a unified understanding of the physical world.

Methodology: VVM-Tuning
The authors propose VVM-Tuning, a training framework designed to equip LMMs with two distinct capabilities: Modality-unaware Perception and Modality-aware Understanding. This is achieved through a pipeline of Fabricated Modality Synthesis and Modality Contexts.

1. Modality Synthesis Pipeline:

   • Image Synthesis: To create diverse training data without requiring real non-RGB datasets, the authors fabricate images from standard RGB sources. This involves converting RGB images to grayscale, applying random cubic spline interpolation to adjust histograms, and mapping them to pseudo-colors using 22 scientific color maps. Additional augmentations (noise, blurring) simulate low-resolution or noisy sensor characteristics.
   • Modality Context Fabrication: Textual prompts are generated to describe the physical meaning of the synthesized modalities (e.g., "warmer colors represent higher temperatures"). For images with segmentation masks, contexts are assembled by linking color maps to objects; for unannotated images, LLMs are prompted to generate contexts based on human-written examples.

2. Training Framework (VVM-Tuning):
   The framework utilizes Supervised Fine-Tuning (SFT) with two sets of disentangled tasks:

   • Modality-unaware Perception: These tasks aim to extract invariant semantics and basic visual elements (photographic perception) regardless of the visual appearance.
     • Semantic Tasks: The model is trained to extract shared underlying semantics between the original RGB image and the fabricated image.
     • Photographic Tasks: The model extracts unique visual clues (e.g., specific color patterns) present in the fabricated image but not in the semantic description.
     • Format: Includes both image descriptions and Visual Question Answering (VQA) tasks.
   • Modality-aware Understanding: These tasks teach the model to connect the synthesized visual appearances with the physical attributes defined in the Modality Context.
     • The model learns to integrate the semantic description, photographic description, and the modality-specific context to answer questions requiring modality knowledge.
     • This enables zero-shot adaptation during inference by utilizing the model's instruction-following and in-context learning abilities.

Key Contributions

• Novel Generalization Approach: The paper is the first to explore generalizing LMMs to unseen visual modalities without in-modality training data, demonstrating that modality synthesis and context can bridge this gap.
• Fabricated Modality Synthesis: A pipeline that generates diverse non-RGB images and corresponding modality contexts from RGB data, allowing for the creation of a general synthetic instruction set.
• Disentangled Training Strategy: A framework that separates "modality-unaware perception" (extracting shared semantics) from "modality-aware understanding" (connecting physical meaning to appearance), preventing the over-alignment of semantics to specific RGB appearances.
• VVM-Bench: A comprehensive benchmark featuring 6 visual modalities (3 real: Thermal, Depth, X-ray; 3 fabricated: Optical Flow, Synthesized 1, Synthesized 2) and 11 VQA tasks to evaluate both perception and understanding capabilities.

Experimental Results
The framework was evaluated on five base models (LLaVA-1.5, Qwen2.5-VL, Qwen3-VL) across the VVM-Bench.

• Perception Improvements: Models trained with VVM-Tuning showed an average 8.2% improvement on the perception subset (tasks without modality contexts) across all tested modalities. The most significant gains were observed in fabricated modalities (e.g., +22.2% for LLaVA-1.5 on Synthesized 1).
• Understanding Improvements: With the inclusion of modality contexts, the models achieved an average 3.8% improvement on the understanding subset.
• Generalization: Despite training on synthetic data, models demonstrated consistent improvements on real-world modalities (Thermal, Depth, X-ray, Optical Flow), confirming that the underlying principles of visual modalities are shared.
• Ablation Studies: Results confirmed that both modality-unaware perception tasks and modality-aware understanding tasks are necessary; removing either component significantly degraded performance, with the perception base being critical for learning adaptability.

Significance and Claims
The authors claim that their work demonstrates a viable path for LMMs to achieve zero-shot generalization to versatile visual modalities. By disentangling invariant semantics from modality-specific appearances and utilizing textual contexts to bridge the gap, LMMs can recognize physical entities and understand modality-specific attributes without being explicitly trained on those specific modalities.

The paper posits that this approach offers a new perspective on machine vision: distinguishing between the "image" (visual appearance) and the "real world" (underlying semantics). The authors suggest that current LMMs often treat the image as the reality, whereas their method encourages a separation that allows for broader generalization. They conclude that with sufficient perception capabilities and modality contexts, LMMs can adapt to unseen visual modalities, and they provide VVM-Bench to facilitate further research in this direction.