💬 NLP

What is Holding Back Latent Visual Reasoning?

This paper reveals that current latent visual reasoning models fail to utilize intermediate visual tokens because existing datasets provide insufficient information to make them necessary and inference-time predictions are inaccurate, indicating that future progress requires both more informative datasets and better token generation.

Original authors: André G. Viveiros, Nuno Gonçalves, André F. T. Martins, Matthias Lindemann

Published 2026-05-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: André G. Viveiros, Nuno Gonçalves, André F. T. Martins, Matthias Lindemann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to solve a complex visual puzzle, like figuring out how a puppy is playing with a toy or rotating a shape in its mind. To help the robot think, you give it a special "thinking space" where it can generate invisible, internal notes (called latent tokens) before it speaks its final answer. The idea is that these notes act like a mental sketchpad, allowing the robot to "imagine" steps in between seeing the picture and giving the answer.

This paper investigates whether these invisible notes actually help the robot, or if the robot is just ignoring them. The researchers found two major problems that are stopping these robots from thinking visually.

The Big Discovery: The Robot is Ignoring the Notes

The researchers tested four different advanced robot models. They did a simple experiment: they replaced the robot's carefully generated "thinking notes" with nonsense gibberish (like random noise or blank spaces).

The result? The robots gave the exact same answers with the same accuracy. It's as if you replaced a student's detailed study notes with a blank piece of paper, and they still got an A. This means the robots aren't actually using the notes to solve the problem; they are just "skipping" them and relying on the original image and text alone.

The authors call this the "Latent Bypass" problem. The robot has a shortcut: it realizes it doesn't need the notes to get the job done, so it ignores them.

Why is the Robot Ignoring the Notes? (Problem #1: Bad Homework)

The paper argues that the fault lies with the training data (the homework the robots were given).

In most current datasets, the "intermediate steps" (the notes) are just cropped zoom-ins of the original picture.

The Analogy: Imagine you are trying to solve a mystery by looking at a crime scene photo. The "thinking notes" you are given are just a zoomed-in picture of the same crime scene. Since you can already see the whole scene clearly in the main photo, the zoomed-in note adds nothing new. You don't need to read the note to solve the mystery, so you ignore it.

The researchers tested this by creating a new "diagnostic" dataset (like a Tetris puzzle). In this new game, the "thinking notes" contained information that could not be seen in the original picture (e.g., a specific rotation rule).

The Result: When the notes contained new, necessary information, the robots finally started using them! They couldn't solve the puzzle without the notes. This proves the robots can use the notes, but only if the notes actually help them.

Why are the Notes so Bad? (Problem #2: The "Blob" Effect)

Even when the robots try to generate their own notes during a test, they fail at the second hurdle. The researchers found that the notes the robots generate are all identical to each other.

The Analogy: Imagine a group of students asked to draw different steps of a story. Instead of drawing unique scenes, they all draw the exact same generic stick figure. No matter what the story is, the drawing is the same.
The Science: The robots' internal "thinking space" collapses into a tiny, narrow area. Instead of creating diverse, useful mental images, they all produce the same bland, uninformative "blob." Because these generated notes are so similar and unhelpful, the robot learns that it's better to ignore them entirely.

The Solution: Two Pillars for Improvement

The paper concludes that for robots to truly "think" visually, we need to fix two things:

Better Homework (Datasets): We need to create training problems where the "thinking notes" contain information that the robot cannot get from the main picture alone. If the notes are just zoomed-in copies, the robot will ignore them. If the notes are essential clues (like a hidden rule), the robot will learn to use them.
Better Thinking (Prediction): We need to teach the robots to generate more diverse and accurate notes. Right now, they are stuck in a rut, producing the same boring notes over and over. They need to learn to create distinct, useful mental sketches.

In short: The technology to let robots "imagine" steps exists, but right now, the training materials are too easy (making the notes useless) and the robots' imagination is too lazy (making the notes identical). Fix the materials and the imagination, and the reasoning will follow.

Technical Summary: What's Holding Back Latent Visual Reasoning?

Problem Statement

Vision-Language Models (VLMs) have achieved strong performance on various visual tasks but continue to struggle with scenarios requiring spatial and compositional reasoning, where solving a problem depends on the internal manipulation of visual representations rather than purely textual descriptions. To address this, recent works have introduced Latent Visual Reasoning, utilizing continuous latent tokens as intermediate "imagination" steps to facilitate visual Chain-of-Thought (CoT) reasoning.

Despite reported performance improvements, it remains unclear what these latent representations actually encode and to what extent they causally influence model predictions. This paper investigates whether current VLMs genuinely rely on latent tokens for reasoning or if they merely bypass them.

Methodology

The authors investigate the causal role of latent tokens across four recent off-the-shelf latent visual reasoning models: LVR, Monet, ILVR, and LanteRn. The study employs a multi-faceted approach:

Intervention Analysis (Causal Testing): The authors replace generated latent tokens ( $\hat{z}$ ) with uninformative "dummy" tokens ( $z'$ ) during inference. Interventions include:
- Random Subregion: Injecting latents derived from unrelated image crops.
- Zeros/Noise: Replacing latents with constant values or Gaussian noise.
- Skip Latents: Disabling the latent pathway entirely by inserting the end token immediately after the start token.
- Oracle Conditioning: Providing ground-truth oracle latent tokens ( $z^*$ ) derived from intermediate images to test if the model can use high-quality information if available.
Diagnostic Dataset Creation: To isolate the impact of training data, the authors constructed specific scenarios:
- Masked Training: Masking relevant regions in the input image so that the task can only be solved using information from the intermediate image (and thus the latent tokens).
- Tetris-like Rotations: A synthetic dataset requiring the model to apply a rotation observed in a source shape to a target shape. Here, intermediate images represent non-trivial transformations (rotations) rather than simple subregions.
Latent Space Analysis: The authors analyzed the distribution of generated latents versus oracle latents using cosine similarity metrics, specifically measuring:
- Retrieval Accuracy: How often a generated latent matches its corresponding oracle latent.
- Unrelated Self-Generated Preference (USP): Whether generated latents are closer to other generated latents than to their ground truth.
- Temporal Collapse: The similarity between consecutive latent tokens over time.

Key Contributions and Results

1. The Latent Bypass Problem

The study reveals that current state-of-the-art models largely ignore latent tokens when producing answers.

Result: Replacing generated latents with uninformative dummy tokens (noise, zeros, or random subregions) resulted in little to no drop in accuracy across all four models on benchmarks like BLINK and V*Bench. In some cases, performance slightly improved.
Oracle Limitation: Even when models were conditioned on high-quality oracle latents (ground truth) during inference, accuracy did not significantly improve over standard inference.
Conclusion: The latent tokens play a minimal causal role in the final prediction. The models effectively bypass the latent reasoning pathway.

2. Training Data as the Root Cause

The authors identify the training data as the primary driver of the bypass problem.

Observation: In existing datasets (e.g., VisCoT, COMT), intermediate images are typically cropped subregions of the original input. Since the full information is already present in the input image, the latent tokens provide no additional necessary information.
Evidence:
- Pause Token Baseline: A baseline model trained without latent tokens (using placeholder "pause" tokens) achieved the same accuracy (79%) as the model using oracle latents on standard data.
- Masked Training: When the input image was masked to hide the solution, forcing reliance on the latent tokens, the model's accuracy dropped significantly if latents were replaced with noise, but remained high with oracle latents.
- Tetris-like Rotations: In a task requiring non-trivial transformation (rotation) not directly visible in the input, models trained on this data learned to rely on latent tokens. Oracle latents yielded 86% accuracy, while interventions dropped performance to chance levels (~30%).
Conclusion: Models only rely on latent tokens when the intermediate steps provide non-trivial, task-relevant information that cannot be easily recovered from the original input.

3. Latent Representation Collapse

Even when models are capable of using latent tokens, they fail to generate high-quality ones during standard inference.

Observation: Generated latent tokens ( $\hat{z}$ ) exhibit extremely high similarity to one another (cosine similarity > 0.8–0.98) but are relatively dissimilar to their corresponding oracle representations ( $z^*$ ).
Retrieval Failure: The models fail to retrieve the correct oracle latent within the top-10 candidates (retrieval accuracy < 10% for most models).
Temporal Collapse: As the reasoning chain progresses, predicted latents collapse into a narrow, highly similar region of the latent space, whereas oracle latents maintain diversity.
Conclusion: Current methods suffer from latent representation collapse, where models generate generic, low-information tokens rather than diverse, discriminative visual abstractions.

Significance and Future Directions

The paper concludes that progress in latent visual reasoning is currently bottlenecked by two fundamental issues:

Dataset Quality: Existing datasets do not incentivize the use of latent tokens because intermediate steps are often redundant with the input image. Future datasets must provide intermediate steps that offer scaffolding for tasks where the solution is not trivially extractable from the raw input (e.g., geometric transformations).
Prediction Precision: Models struggle to predict diverse and accurate latent tokens, leading to representation collapse. Future work must focus on improving the generation of these tokens to match the quality of oracle representations.

The authors emphasize that while latent visual reasoning holds promise for spatial and compositional tasks, current implementations are largely ineffective due to these training and generation flaws. They provide the code, models, and diagnostic datasets (LanteRn) to facilitate further research in this direction.