🤖 AI

Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Models

This paper proposes a self-captioning framework with a Multimodal Interaction Gate to convert unique modality information into redundant shared information, thereby significantly reducing hallucinations and improving the robustness of vision-language models against corrupted inputs.

Original authors: Yuriel Ryan, Hei Man Ip, Adriel Kuek, Paul Pu Liang, Roy Ka-Wei Lee

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuriel Ryan, Hei Man Ip, Adriel Kuek, Paul Pu Liang, Roy Ka-Wei Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "One-Sided" Student

Imagine you are teaching a robot student (a Vision Language Model) to understand the world. This student has two eyes (to see images) and a brain (to read text).

Currently, most training methods are like a strict teacher who says: "Look at this picture of a cat. The answer is 'cat'. Don't read the text; just look at the picture."

The paper argues that this approach makes the student too dependent on the picture. If the picture is blurry, dark, or confusing (corrupted), the student panics and starts guessing wildly (hallucinating). They might say, "That's a dog!" just because the picture is fuzzy, because they never learned to cross-check the picture with the text.

The Core Idea: The "Safety Net"

The authors hypothesize that the robot needs a safety net. In information theory, this is called redundancy.

Think of redundancy like a pilot and a co-pilot in a plane.

Unique Information: The pilot sees the runway; the co-pilot reads the weather report. They have different, exclusive info.
Redundant Information: Both the pilot and the co-pilot can see the runway and both can read the weather report. They are saying the same thing in two different ways.

The paper suggests that if the robot learns from data where the image and the text both say the same thing (redundancy), it becomes much harder to fool. If the image gets corrupted (like a foggy runway), the robot can still rely on the text to know what's happening.

The Solution: The "Self-Captioning" Workflow

The researchers created a tool called the Multimodal Interaction Gate (MI Gate). Here is how it works, step-by-step:

The Audit: The system scans a dataset of images and text. It looks for "Unique Visual" moments—cases where the picture tells a story the text doesn't mention.
- Analogy: Imagine a photo of a dog chasing a ball. The text just says, "A dog is running." The picture shows the ball. The picture has "unique" info the text lacks.
The Transfer: The system takes those pictures and asks the robot to write a description (a caption) for them.
The Merge: It adds that new description to the original text.
- Result: Now, the text says, "A dog is running. [It is chasing a ball]."
- The Magic: The information about the ball is no longer only in the picture. It is now in both the picture and the text. The "Unique" info has been converted into "Redundant" info.

Why This Matters (The Results)

The paper tested this by training robots with this new "safety net" data and then trying to trick them.

The Test: They showed the robots blurry, noisy, or corrupted images.
The Outcome:
- The robots trained with the "safety net" made 38.3% fewer mistakes when the images were bad.
- They were 16.8% more consistent (they didn't flip-flop between answers).
- They learned to use both their eyes and their brain more equally, rather than ignoring the text.

The Catch: The "Too Much of a Good Thing" Rule

The paper also found a limit. You can't just turn everything into redundancy.

The Analogy: Imagine a puzzle where the image and text work together to solve a riddle (Synergy). If you force the text to describe the image perfectly, you might ruin the puzzle. The robot stops learning how to combine clues and just starts reading the answer.
The Finding: If they captioned every single image (100%), the robot actually got worse at some tasks. It needed a mix: some data where the image and text were different (to learn to combine them) and some where they were the same (to build the safety net).

Summary

The paper proposes a simple fix for robots that hallucinate: Don't just show them pictures; make sure the text describes the pictures too.

By automatically generating captions for images and adding them to the text, they create a "double-check" system. This makes the robot robust against bad images because it has learned that if the picture is unclear, the text can save the day, and vice versa. It turns a fragile, one-sided learner into a resilient, double-checking expert.

Technical Summary: Self-Captioning Multimodal Interaction Tuning

Problem Statement

Vision Language Models (VLMs) currently face significant challenges regarding hallucinations and robustness when processing ambiguous or corrupted modalities. While existing approaches attempt to mitigate these issues by training VLMs to be "grounded" on visual content using instruction tuning datasets, these datasets often prioritize visual grounding to the extent that they eliminate redundant interactions (shared information between modalities). The authors hypothesize that this lack of redundancy leaves models vulnerable; without shared information to compensate for an impaired modality, the model cannot effectively resolve ambiguities or recover from corrupted inputs. Current data curation strategies rely heavily on intuition and heuristics rather than systematic insights derived from multimodal interaction theory.

Methodology

The paper proposes a systematic data augmentation strategy centered on the Multimodal Interaction (MI) Gate, a mechanism designed to convert unique interactions (information exclusive to one modality) into redundant interactions (shared information).

Theoretical Framework

The methodology is grounded in Pointwise Partial Information Decomposition (PPID), which decomposes the information provided by modalities ( $X_V$ for visual, $X_T$ for text) about a target task ( $Y$ ) into three components:

Redundant ( $R$ ): Shared information available in both modalities.
Unique ( $U$ ): Information exclusive to a single modality ( $U_V$ or $U_T$ ).
Synergistic ( $S$ ): Emergent information only available when both modalities are observed jointly.

The core hypothesis is that increasing $R$ in training data allows the model to utilize shared information to cover for ambiguous or corrupted inputs, thereby improving robustness.

The MI Gate Workflow

The proposed Self-Captioning Multimodal Interaction Tuning workflow operates as follows:

Interaction Estimation: Using classifiers and entropy estimators, the system quantifies $R$ , $U_V$ , $U_T$ , and $S$ for each sample in a dataset.
Sample Selection: The system identifies samples where Unique Visual ( $U_V$ ) is the dominant interaction.
Interaction Transfer: For selected samples, the image is captioned by a VLM. This caption is appended to the original text instruction.
- This process transfers information that was previously unique to the image ( $U_V$ ) into the text modality, creating a shared, redundant signal ( $R$ ).
- The system employs a threshold ( $\tau$ ) to control the percentage of samples captioned.
Filtering: To avoid disrupting synergistic structures (where the answer relies on the interplay of both modalities), the gate bypasses captioning for samples where Synergy ( $S$ ) is dominant.
Training: The augmented dataset, now containing higher levels of redundant information, is used for Supervised Fine-Tuning (SFT) of VLMs (specifically SmolVLM and LLaVA-OneVision families).

Key Contributions

Systematic Analysis of Redundancy: The authors utilized the PPID framework to demonstrate that increasing redundant interactions in instruction datasets can theoretically and empirically produce more robust VLMs against ambiguous and corrupted modalities.
The MI Gate Mechanism: They operationalized this analysis into a data-centric mechanism that systematically increases redundancy by transferring unique visual contributions into redundant interactions via self-captioning.
Empirical Validation: The study provides evidence that increasing redundant information reduces visual-induced errors and improves model consistency.

Experimental Results

The authors evaluated the approach on various model families (SmolVLM and LLaVA-OneVision) across different parameter sizes and datasets (e.g., Hateful Memes, GQA, Cauldron).

Reduction in Visual Hallucinations: Increasing redundancy reduced visual-induced (VI) hallucinations by up to 38.3% (specifically in SmolVLM models with 50% redundancy injection).
Improved Consistency: Model consistency scores improved by 16.8% on average.
Robustness to Corruption: Models trained with increased redundancy demonstrated higher performance stability ( $\Delta P$ ) when facing corrupted inputs (both image noise and text corruption), particularly in image corruption scenarios.
Trade-offs: The authors observed a trade-off: while visual-induced errors decreased, language-induced (LI) and mixed errors increased slightly. This suggests the model relies more heavily on both modalities, making it more susceptible to text-based ambiguities but significantly more resilient to visual degradation.
Interaction Transfer Validity: Experiments confirmed that captioning successfully converts $U_V$ to $R$ without introducing significant noise, provided the captioning model is capable and the transfer is not forced on highly synergistic tasks (where it could disrupt the learning of inter-dependencies).

Significance and Claims

The paper claims to offer a potential solution to the heuristic-based approaches currently used in curating instruction datasets. By shifting from intuition-driven data augmentation to an interaction-aware design, the authors demonstrate that:

Redundancy is a trainable asset: Contrary to the convention of minimizing redundancy to force visual grounding, explicitly amplifying exploitable redundancies creates a more resilient system.
Systematic Improvement: The MI Gate provides a reproducible framework to adjust multimodal interactions, moving beyond trial-and-error data curation.
Feasibility of Interaction Transfer: The work proves that unique information can be systematically converted into shared information to enhance model reliability against degraded inputs.

The authors maintain a modest stance, acknowledging limitations such as the unidirectional nature of the transfer (visual to text) and the reliance on estimators trained on discrete tasks. They conclude that while net benefits are observed in robustness against ambiguous modalities, future work must further investigate the impacts of multimodal interactions across different task categories and model architectures.