💬 NLP

Reasoning in the Dark: Interleaved Vision-Text Reasoning in Latent Space

This paper proposes Interleaved Vision-Text Latent Reasoning (IVT-LR), a novel framework that injects implicit visual and textual information into the latent space to enable efficient, annotation-light multimodal reasoning, achieving significant gains in both accuracy and inference speed compared to existing explicit methods.

Original authors: Chao Chen, Zhixin Ma, Yongqi Li, Yupeng Hu, Yinwei Wei, Wenjie Li, Liqiang Nie

Published 2026-01-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chao Chen, Zhixin Ma, Yongqi Li, Yupeng Hu, Yinwei Wei, Wenjie Li, Liqiang Nie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a complex puzzle, like a tricky riddle that involves both a picture and a sentence.

The Old Way: The "Talk-It-Out" Method
Currently, most smart AI models solve these puzzles by "talking out loud" as they think. They look at the picture, then they write down a long list of sentences explaining what they see, then they look at the picture again, write more sentences, and finally give the answer.

The Problem: This is like trying to solve a math problem by writing out every single thought in a diary before you can write the final number. It takes a lot of time (slow), requires someone to write out thousands of these "diaries" to teach the AI (expensive labor), and the AI has to read and write all those words just to get to the point.

The New Way: "Reasoning in the Dark" (IVT-LR)
This paper introduces a new method called IVT-LR (Interleaved Vision-Text Latent Reasoning). Think of this as the AI learning to think silently.

Instead of writing down its thoughts, the AI keeps its reasoning process entirely inside its own "brain" (the latent space). It doesn't generate words or draw pictures while it's thinking. It just processes the information internally.

Here is how the new method works, using a simple analogy:

1. The "Ghost" and the "Spotlight"

In the old way, the AI had to describe the picture in words. In this new way, the AI uses two invisible tools to think:

The Ghost (Latent Text): Instead of writing "I see a red ball," the AI holds onto a "ghost" of its previous thought. It's a hidden signal that carries the logic of what it just thought, without needing to say it out loud.
The Spotlight (Latent Vision): Instead of describing the whole picture, the AI uses a mental spotlight. It quickly scans the image and picks only the most important tiny pieces (like a specific part of a diagram) to focus on for that specific step of thinking.

2. The "Silent Conversation"

The AI mixes these two invisible tools together. It takes the "Ghost" of its last thought and combines it with the "Spotlight" on the most important part of the image. It does this step-by-step, entirely in the dark (without generating any visible text or images), until it is ready to shout out the final answer.

3. The Training: Learning to Whisper

How do you teach an AI to think silently? You can't just tell it to stop talking; it needs to learn how to do it.
The authors used a progressive training strategy, which is like teaching a child to swim:

Stage 1: The AI learns to solve the puzzle by talking out loud (writing all the steps).
Stage 2: The teacher says, "Okay, for the first step, don't write it down. Just think it."
Stage 3 & 4: The teacher gradually asks the AI to stop writing down more and more steps, replacing them with silent thinking, until the AI is solving the whole puzzle in its head and only speaks the final answer.

Why is this a big deal?

The paper claims this method is a massive upgrade for three reasons:

Speed: Because the AI isn't wasting time writing out long explanations, it solves problems 5 times faster.
Smarter Thinking: It can mix pictures and words in its "brain" more naturally, rather than forcing the picture into clumsy words.
Less Work: You don't need humans to write out thousands of detailed "thought processes" to teach the AI. The AI learns to think silently on its own.

The Results
When they tested this on difficult science and logic puzzles (using datasets called M3CoT and ScienceQA), the new method got more correct answers (about 5% better) and did it much faster than the old "talk-it-out" methods.

In a nutshell: The paper teaches AI to stop "narrating" its thoughts and start "thinking" silently, using a mix of hidden logic and focused visual attention, making it faster, smarter, and cheaper to train.

Technical Summary: Reasoning in the Dark: Interleaved Vision-Text Reasoning in Latent Space (IVT-LR)

Problem Statement

Current multimodal reasoning methods for Multimodal Large Language Models (MLLMs) face significant limitations regarding annotation costs and inference efficiency. Existing approaches generally fall into two categories:

Text-only reasoning: MLLMs perform reasoning using only textual rationales before generating an answer. While effective, this often fails to leverage visual information during the intermediate thought process.
Vision-text involved reasoning: These methods interleave visual information with textual rationales (e.g., generating image patches, zooming in on regions, or sketching images). While this improves performance on complex tasks, it relies heavily on labor-intensive vision-text annotations and introduces significant inference latency due to the generation of long chains of explicit tokens.

The authors identify a need for a paradigm that retains the benefits of multimodal reasoning (integrating visual and textual cues) while eliminating the overhead of explicit intermediate generation.

Methodology: Interleaved Vision-Text Latent Reasoning (IVT-LR)

The paper proposes IVT-LR, a framework that enables MLLMs to perform reasoning entirely within the latent space, interleaving visual and textual modalities without generating explicit intermediate tokens.

Core Mechanism

In IVT-LR, each reasoning step is represented by a multimodal latent vector composed of two implicit parts:

Latent Text: Instead of generating an explicit text token for the previous step, the model uses the hidden state ( $h_{t-1}$ ) from the previous layer as the input for the current step. This preserves rich intermediate reasoning logic in a continuous space.
Latent Vision: To incorporate visual context dynamically, the model selects a fixed number ( $k$ ) of image embeddings based on attention scores. Specifically, the sum of attention weights across all layers is used to identify the $k$ most relevant image embeddings. These selected embeddings are concatenated with the latent text hidden state.

The model fuses these components ( $[h_{t-1}, z_{selected}]$ ) to compute the next token distribution, effectively performing step-wise reasoning without outputting intermediate text or images.

Training Strategy

To enable MLLMs to adapt to this latent reasoning paradigm, the authors introduce a progressive multi-stage training strategy:

Stage 0: The model is trained with standard Chain-of-Thought (CoT) supervision, where all reasoning steps are explicitly generated.
Progressive Substitution: Over $N$ subsequent stages, the model gradually replaces explicit reasoning steps with latent reasoning steps (denoted by a special <latent> token). In each stage, one additional explicit step is swapped for a latent step.
Loss Function: Training utilizes Negative Log-Likelihood (NLL) loss. Crucially, supervision is applied only to the remaining explicit reasoning steps and the final answer. The latent reasoning steps and question tokens are masked out. This prevents the model from being forced into a strict linear path of alignment with explicit rationales, allowing it to internalize reasoning trajectories in the latent space while still being driven toward correct final predictions.

Inference Process

During inference, the model appends a fixed number of <latent> tokens after the question and image inputs. The reasoning process occurs entirely within the latent space, and no intermediate explicit tokens are generated before the final answer is produced.

Key Contributions

First Unified Multimodal Latent Reasoning Framework: IVT-LR is the first approach to unify textual and visual representations entirely within the latent space, eliminating the need for intermediate explicit text or image generation.
Data and Computation Efficiency: The method introduces a training paradigm that does not require explicit annotations for intermediate visual reasoning steps. By reasoning in the latent space, it drastically reduces the number of autoregressive steps required for inference.
State-of-the-Art Performance: The approach achieves new state-of-the-art results in accuracy on challenging benchmarks while significantly improving inference efficiency (measured by fewer autoregressive steps and lower latency).

Experimental Results

The authors evaluated IVT-LR on two benchmarks: M3CoT (multimodal chain-of-thought) and ScienceQA. Experiments were conducted using Qwen2-VL-7B and Chameleon-7B backbones.

Accuracy: IVT-LR achieved an average performance increase of 5.45% in accuracy compared to existing approaches.
- On M3CoT (Qwen2-VL), IVT-LR reached 71.8% accuracy, outperforming the previous best (Chain-of-Focus at 64.3%).
- On ScienceQA (Qwen2-VL), it achieved 94.6% accuracy, surpassing Chain-of-Focus (91.2%).
Efficiency: The method demonstrated a speed increase of over 5 times compared to existing methods.
- Autoregressive Steps: IVT-LR reduced the average number of autoregressive steps to 10.0 (M3CoT) and 11.0 (ScienceQA), compared to 73.9–185.7 steps for other methods.
- Latency: Average generation time dropped significantly (e.g., from 2.63s to 0.65s on M3CoT with Qwen2-VL).

Significance

The paper claims that IVT-LR represents a shift toward "Reasoning in the Dark," where the complex interplay of visual and textual information occurs internally within the model's hidden states. By decoupling the reasoning process from the need for explicit, observable intermediate outputs, the method addresses the bottlenecks of annotation intensity and inference latency inherent in current multimodal reasoning systems. The authors posit that this approach unlocks the potential of latent reasoning to handle rich, multimodal information more efficiently than traditional token-based generation.