💬 NLP

Bridging the Missing-Modality Gap: Improving Text-Only Calibration of Vision Language Models

The paper proposes the Latent Imagination Module (LIM), a lightweight cross-attention mechanism that predicts imagined latent embeddings from text to restore the accuracy and calibration of Vision-Language Models when deployed without their original visual inputs.

Original authors: Mingyeong Kim (Kim Jaechul Graduate School of AI, KAIST), Jungwon Choi (Kim Jaechul Graduate School of AI, KAIST), Chaeyun Jang (Kim Jaechul Graduate School of AI, KAIST), Juho Lee (Kim Jaechul Gradua

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mingyeong Kim (Kim Jaechul Graduate School of AI, KAIST), Jungwon Choi (Kim Jaechul Graduate School of AI, KAIST), Chaeyun Jang (Kim Jaechul Graduate School of AI, KAIST), Juho Lee (Kim Jaechul Graduate School of AI, KAIST)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Multitasking Chef" Who Forgets How to Cook Alone

Imagine a world-class chef (the Vision-Language Model, or VLM) who was trained in a high-end kitchen where they always had fresh ingredients (images) and a recipe card (text) in front of them. They learned to cook amazing dishes by looking at the ingredients and reading the instructions together.

Now, imagine you ask this chef to cook a dish using only the recipe card, with no ingredients in sight.

The paper finds that when you do this, two bad things happen:

The food tastes worse: The chef makes mistakes because they are used to seeing the ingredients.
The chef gets dangerously overconfident: Even though the food is bad, the chef is 100% sure it's perfect. They don't realize they are missing crucial information.

The researchers discovered that simply describing the missing ingredients in more detail (e.g., "Imagine a red tomato") doesn't fix the chef's confidence. The chef still feels lost because their brain is wired to see, not just to read.

The Discovery: It's Not Just About Missing Words

The team ran several experiments to prove this:

The "Description" Test: They replaced photos with detailed text descriptions. The chef's accuracy dropped, and their confidence became wildly unreliable.
The "Fake Photo" Test: They used a computer to generate a fake photo based on the text description and showed it to the chef. Suddenly, the chef's performance and confidence improved! This proved that the chef needs a visual signal, even if it's just a fake one, to feel "grounded."
The "Back to Basics" Test: They compared the chef to a pure text-only chef (a standard Language Model). Surprisingly, the multimodal chef did worse than the text-only chef when no pictures were shown. The multimodal chef couldn't just "switch off" their visual brain; it was actually confused by the lack of images.

The Solution: The "Imagination Module" (LIM)

The researchers proposed a clever fix called the Latent Imagination Module (LIM).

Instead of asking the chef to wait for a real photo or a computer to generate a full, high-definition image (which is slow and expensive), LIM acts like a mental sketch artist.

How it works: When the chef gets a text-only question, LIM instantly creates a "mental image" in the chef's brain. It doesn't draw a picture with pixels (like a real photo); instead, it creates a set of abstract "thought tokens" that look exactly like the visual data the chef is used to seeing.
The Analogy: Think of it like a movie director. If an actor (the AI) needs to react to a dragon but there is no dragon on set, the director doesn't build a massive, expensive dragon prop. Instead, the director whispers, "Imagine a giant, scaly beast right there," and hands the actor a specific cue card that triggers the exact same emotional reaction as seeing the real dragon.

Why This is Better Than Other Methods

It's Fast: Generating a real image takes a long time (like building a full prop). LIM just creates the "mental cue" instantly. The paper says it's 12 times faster than generating a real image.
It's Smarter: They tested if just filling the empty spot with anything (like a blank white image or random noise) would help. It didn't. The "mental sketch" had to be specifically trained to match the text. It's not just about having something there; it's about having the right kind of imagined thing.
It Works Everywhere: They tested this on questions the chef had never seen before (unseen tasks), and it still worked. The chef became more accurate and, most importantly, stopped being overconfident when they were wrong.

The Big Takeaway

The paper concludes that when we force a "multimodal" AI (one that sees and reads) to work with only text, it gets confused and unreliable. We can't just tell it to "use its language brain."

Instead, we need to give it a virtual visual signal—a "latent imagination"—that tricks its brain into thinking it has the visual context it needs. This makes the AI smarter, more accurate, and much more honest about how sure it is of its answers.

Technical Summary: Bridging the Missing-Modality Gap in Vision-Language Models

Problem Statement

Vision-Language Models (VLMs) are increasingly deployed in real-world scenarios where they must process text-only inputs, despite being trained on multimodal (image-text) data. The authors identify a critical failure mode in these settings: when the visual modality is absent, VLMs suffer not only from reduced predictive accuracy but, more significantly, from severe miscalibration.

Empirical analysis reveals that text-only VLMs do not simply "fall back" to the behavior of their underlying language backbones (e.g., LLaVA does not perform like its base Vicuna). Instead, they exhibit systematic overconfidence, where predicted confidence scores remain high despite a sharp drop in empirical accuracy. This phenomenon persists even when the missing visual content is replaced by high-fidelity textual descriptions, suggesting the issue is not merely a deficit of semantic information but a cognitive rupture caused by a mismatch between the multimodal training distribution and the text-only inference dynamics.

Methodology: Latent Imagination Module (LIM)

To address the missing-modality gap without the computational cost of pixel-level image generation, the authors propose the Latent Imagination Module (LIM).

Core Architecture

LIM is a lightweight, cross-attention module designed to predict "imagined" latent embeddings directly from textual input. These embeddings are injected into the frozen VLM backbone to replace the missing visual tokens.

Input: Text embeddings ( $E_T$ ) extracted from the VLM's text layer.
Mechanism: The module initializes a set of learnable latent queries ( $E_Q$ ) corresponding to the expected number of visual tokens. These queries are refined through a stack of cross-attention layers that attend to the text embeddings.
Output: Synthesized visual features ( $\hat{z}_V$ ) that are compatible with the VLM's vision tower projection layer.
Inference: The synthesized features are injected into the input sequence in place of the missing image, allowing the frozen VLM to proceed with inference as if a visual signal were present.

Optimization Strategy

A key design choice is the training objective. Unlike approaches that minimize the distance between predicted embeddings and ground-truth image embeddings (which can lead to "blurry" averages due to the one-to-many nature of text-to-image mapping), LIM is trained end-to-end using a task-oriented objective.

The module minimizes the negative log-likelihood of the correct answer given the text and the imagined features: $-\log P(y | x_T, \hat{z}_V)$ .
This allows the model to synthesize any "sufficient" visual representation that aids the decision boundary, rather than forcing a pixel-perfect reconstruction.
The VLM backbone remains frozen; only LIM parameters are updated.

Key Contributions

Identification of the Missing-Modality Gap: The authors frame the miscalibration in text-only VLMs not as a simple data deficit but as a cognitive rupture where the model fails to utilize its language backbone effectively, leading to severe overconfidence.
Latent Imagination Module (LIM): A novel architecture that bridges the modality gap by inferring task-relevant latent embeddings from text via cross-attention, avoiding expensive pixel-level generation.
End-to-End Latent Imputation: The demonstration that a task-oriented latent completion strategy significantly improves both accuracy and calibration across inherently text-only tasks and scenarios where images are partially missing.

Experimental Results

The authors evaluated LIM on the LLaVA-1.5-7B model across three experimental families:

1. In-Domain Text-Only Calibration

On the MMLU+ARC text-only suite:

Accuracy: Improved from 39.77% (baseline) to 62.23%.
Calibration: Expected Calibration Error (ECE) dropped drastically from 0.4202 to 0.0374.
Comparison: LIM outperformed temperature scaling (a standard post-hoc calibration method), which reduced ECE to 0.095 but did not improve accuracy.
Efficiency: LIM reduced inference latency by 12.26× compared to diffusion-based image regeneration (0.205s vs. 2.52s per sample) while maintaining similar VRAM usage.

2. Generalization to Unseen Tasks

LIM was tested on eight unseen text-only benchmarks (e.g., SST-2, COLA, LOGIQA, CommonSenseQA) without further tuning.

Results: The module consistently improved calibration and often accuracy across heterogeneous NLP tasks. For example, on COLA, accuracy increased from 32.50% to 68.00%, and ECE dropped from 0.6119 to 0.1047.
Implication: LIM learns a transferable correction for the missing-modality gap rather than overfitting to the training distribution.

3. Robustness to Missing Images in Paired Tasks

In a simulation where images were randomly dropped from the ScienceQA dataset:

Baseline: As the drop rate increased, accuracy declined and overconfidence worsened.
LIM: Maintained robustness. At a 100% drop rate, LIM achieved accuracy comparable to the full-image baseline (56.77% vs. 54.54%) with significantly better calibration (ECE 0.0936 vs. 0.3054).

Ablation Studies

Controlled experiments confirmed that the improvements are not due to simply filling missing token slots with arbitrary embeddings. Substitutes such as zero vectors, whitespace tokens, random noise, or blank images failed to reproduce the gains, and in some cases degraded performance. This validates that the benefits stem from structured, text-conditioned latent completion.

Significance and Claims

The paper claims that latent modality completion is a practical and principled approach for reliable VLM inference under missing-modality conditions.

Reliability: The primary contribution is stabilizing confidence predictions. By aligning text-only inference with the multimodal training distribution via latent imagination, LIM prevents the dangerous overconfidence that plagues current VLMs in text-only settings.
Architectural Simplicity: Unlike routing text queries to a separate LLM (which introduces system complexity and potential behavioral inconsistencies), LIM augments the unified VLM with a lightweight module, preserving a single model architecture.
Efficiency: The approach offers a computationally efficient alternative to generative augmentation (diffusion), making it viable for real-time deployment.

The authors conclude that the failure of VLMs in text-only settings is a systematic mismatch in internal representations, and that bridging this gap via latent imagination is essential for deploying trustworthy multimodal systems.