🤖 machine learning

Scaffolding Minds: Optimizing Latent Visual Target Representations for Multimodal Reasoning

The paper proposes "Scaffolding Minds," a method that optimizes multimodal reasoning by introducing a dedicated scaffolding encoder for better latent target representations in supervised fine-tuning and learning both mean and variance for the reinforcement learning sampler to enable effective exploration, thereby significantly outperforming existing baselines across various visual reasoning tasks.

Original authors: Haoqiang Kang, Yinpeng Chen, Luyang Liu, Jesper Sparre Andersen, Abhijit Ogale, Baochen Sun, Lichan Hong, Ed H. Chi

Published 2026-08-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haoqiang Kang, Yinpeng Chen, Luyang Liu, Jesper Sparre Andersen, Abhijit Ogale, Baochen Sun, Lichan Hong, Ed H. Chi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Artificial intelligence has long struggled to see the world the way humans do. When a computer looks at a picture, it can often identify objects, but it frequently fails to understand the relationships between them or to reason through a complex visual puzzle. For years, researchers have tried to fix this by teaching machines to "think" out loud, forcing them to write down a step-by-step explanation in text before giving an answer. While this helps, it often forces the machine to translate rich, detailed visual information into clumsy words, losing important nuances along the way. A newer approach attempts to solve this by letting the machine keep its thoughts in a hidden, internal language—a continuous stream of data that exists only inside the computer's brain. This method, known as latent reasoning, allows the model to hold onto visual details without having to describe them in sentences. However, even this advanced technique has hit a wall: the internal thoughts the machine generates are often based on generic, pre-made tools that were not designed for the specific problem at hand, and the machine struggles to explore different ways of thinking to find the best solution.

A team of researchers at Google DeepMind and the University of California, San Diego, has developed a new framework called Scaffolding Minds to overcome these hurdles. Their work focuses on two specific stages where these internal reasoning systems typically fail. First, during the initial learning phase, the machine is usually taught to mimic the output of a standard, off-the-shelf vision encoder—a tool trained on millions of general images to recognize cats, cars, and landscapes. The problem is that this generic tool is not optimized for the specific reasoning task the machine is trying to solve. It might preserve irrelevant visual details while missing the crucial evidence needed to solve a puzzle. Second, when the machine tries to improve itself through trial and error, existing methods often force it to stick rigidly to a single path of thought or only allow it to adjust its written answers, leaving the internal visual reasoning unchanged. This prevents the machine from discovering new, better ways to solve the problem.

To fix this, the researchers introduced a two-part system that acts like a custom-built guide for the machine's internal thoughts. In the first stage, instead of relying on a generic tool, they trained a dedicated "scaffolding encoder." Think of this as a specialized tutor that learns exactly what kind of internal notes are most helpful for the specific task, rather than just copying a general dictionary. This tutor takes a helper image—a visual aid provided only during training—and converts it into a set of optimized internal tokens that are perfectly tuned for the reasoning challenge. Once the machine learns to generate these optimized notes on its own, the tutor is discarded, and the machine is ready to work without help.

In the second stage, the researchers replaced the rigid rules that usually govern the machine's internal thoughts with a flexible, learning-based sampler. Instead of forcing the machine to stay on a single, predetermined track, this new method allows the machine to sample different variations of its internal thoughts during practice. It learns to adjust the "mean" and "variance" of its thinking, effectively exploring a wider range of possibilities to see which internal path leads to the correct answer. This exploration is guided by rewards, encouraging the machine to find the most effective reasoning trajectory rather than just settling for the first one it finds.

The results of this approach were tested on a series of challenging visual reasoning tasks, including a spatial planning game called FrozenLake, where a character must navigate a grid of ice and holes to reach a goal. On the standard 8x8 grid, the new method improved the machine's accuracy by 9.5 percent compared to the strongest previous attempts. As the puzzles became more difficult, with grids expanding to 32x32, the advantage grew significantly, with the new system outperforming the best prior methods by 19 percent. The researchers also tested the system on nine other visual reasoning benchmarks that require fine-grained observation and comparison. Across all these tests, the method consistently improved performance, raising average accuracy by 5.2 percent.

The study explicitly ruled out the idea that simply generating new images during the reasoning process is necessary for success. Other methods that try to draw intermediate pictures to help the machine think often suffer from high computational costs and slower speeds. The Scaffolding Minds approach achieved superior results without generating any extra images, proving that optimizing the internal, hidden representation of the visual data is more effective than creating visible intermediate steps. The researchers found that the two improvements—the custom-trained tutor for the internal notes and the flexible exploration of those notes—worked best when used together. Without the custom tutor, the flexible exploration could not reach its full potential, and without the exploration, the custom tutor could not refine its path to the best solution.

This work suggests that the quality of the internal target the machine aims for is just as important as the method it uses to get there. By teaching the machine to generate its own optimized internal notes and then allowing it to explore different variations of those notes, the researchers have created a system that can reason through complex visual problems with greater precision and reliability. The findings indicate that future advances in visual reasoning may depend less on making machines "see" more or "speak" better, and more on teaching them how to structure their hidden, internal thoughts in a way that is perfectly aligned with the task at hand.

Technical Summary: Scaffolding Minds

Problem Statement

Latent visual reasoning has emerged as a compelling solution for multimodal reasoning, allowing Vision-Language Models (VLMs) to "think" in both text and vision by interleaving continuous latent tokens into the reasoning chain. Current methods typically follow a two-stage training paradigm: (1) Supervised Fine-Tuning (SFT), where a "helper image" is encoded into latent tokens using a frozen, off-the-shelf vision encoder to serve as a target; and (2) Reinforcement Learning (RL), where the model is refined using reward feedback.

The authors identify two critical limitations in this existing framework:

Suboptimal SFT Targets: The latent targets in the SFT stage are derived from frozen, general-purpose vision encoders. These encoders are not optimized for the specific downstream reasoning task, often preserving perceptual content irrelevant to the task while failing to emphasize the intermediate evidence required for reasoning. Consequently, the model is supervised toward a convenient but suboptimal target.
Limited RL Exploration: Existing RL methods either ignore the latent block or apply deterministic regularization (e.g., fixed-variance Gaussian constraints) to the latent output. These approaches do not sample alternative latent actions, thereby preventing the exploration of diverse latent reasoning trajectories. This lack of exploration leads to small or unstable performance gains.

Methodology: Scaffolding Minds

The paper proposes Scaffolding Minds, a two-stage framework designed to address these limitations by optimizing both the latent target representation and the exploration mechanism.

Stage 1: SFT via Scaffolding Encoder

Instead of relying on a frozen off-the-shelf encoder, the authors introduce a learnable Scaffolding Encoder ( $\psi$ ).

Scaffolding Phase: A dedicated encoder is trained end-to-end to map the helper image $h$ to a target latent block $z^*$ . This encoder is optimized directly through the downstream task loss (cross-entropy on the answer tokens) while the base VLM is frozen. This ensures $z^*$ captures exactly the reasoning-sensitive features the VLM needs.
Generation Phase: The base VLM is fine-tuned to generate its own latent block $z$ from the input $x$ alone, supervised by an $L_2$ matching loss against the frozen target $z^*$ . The helper image is discarded at inference.
Architecture: The scaffolding encoder consists of a trainable copy of the VLM's vision encoder followed by a cross-attention pooling module.

Stage 2: Scaffolding RL

To enable exploration in the latent space, the authors replace deterministic regularization with Scaffolding RL, which employs an explicit, input-adaptive Gaussian policy over residual latent actions.

Residual Action Sampling: The model generates a base latent prior $z_\theta$ . Two lightweight MLP heads predict an input-adaptive mean $\mu(z_\theta)$ and variance $\sigma^2(z_\theta)$ for a Gaussian distribution.
Exploration: During rollout, the model samples a residual adjustment $\Delta z \sim \mathcal{N}(\mu, \sigma^2)$ and applies it to the prior ( $z_\theta + \Delta z$ ). This sampled action is evaluated under both the current and old policies to compute a reward-guided advantage.
Stability: The mean head is zero-initialized to center the initial distribution on the prior, while the variance head starts at small positive values to ensure stable initial exploration.

Key Contributions

Optimized Latent Targets: The paper introduces a scaffolding encoder that learns latent targets end-to-end through the downstream task, moving away from the suboptimal, frozen features of prior work.
Stochastic Latent Exploration: It proposes Scaffolding RL, which replaces deterministic regularization with a learned Gaussian policy that samples residual latent actions, enabling direct exploration of alternative reasoning trajectories.
Complementarity: The authors demonstrate that the optimized target (Stage 1) and the exploration mechanism (Stage 2) are complementary; the scaffolding encoder provides a high-accuracy starting point, while Scaffolding RL refines this target through reward-driven exploration.

Experimental Results

The method was evaluated on FrozenLake (spatial planning) and nine visual-centric reasoning benchmarks.

FrozenLake (Spatial Planning):
- Scaffolding Minds improved the strongest prior latent baseline (VaLR) by +9.5% on average across grid sizes 8×8 to 32×32.
- The performance gap widened with task difficulty, reaching +19% on the most challenging 32×32 grids.
- Compared to the SFT baseline, the full method achieved a +23.0% absolute improvement.
- Ablation studies confirmed that target optimization (tuning the vision encoder) drove the majority of gains (+10.0% over just changing the objective), and that sampling learned adjustments in RL was essential for stable gains (+3.0% over deterministic methods).
Visual-Centric Reasoning Benchmarks:
- Across nine benchmarks (including V★, BLINK, MMVP, CV-Bench, etc.), the method achieved an average accuracy increase of +5.2% over the strongest prior latent baseline.
- Gains were most pronounced on tasks requiring fine-grained spatial and visual distinctions (e.g., +20.7% on MMVP, +17.6% on MME-RealWorld-Lite).
- The method outperformed "think-with-image" systems that generate explicit intermediate images, achieving strong results with near-base-VLM inference costs.
Efficiency:
- The method adds only 4.6% inference latency overhead compared to the base VLM, significantly lower than image-generation methods which incur >100% overhead.

Significance and Claims

The paper claims that Scaffolding Minds demonstrates that the quality of the latent target and direct exploration in latent space are both critical for effective latent visual reasoning. By treating the latent target as an optimized component of the system rather than a fixed feature, and by enabling stochastic exploration of latent trajectories, the method achieves substantial gains over strong baselines without the computational cost of generating intermediate images. The authors position this work as a step toward more efficient and capable multimodal reasoning, showing that explicit image generation is not strictly necessary if the latent space itself is properly optimized and explored.

The paper acknowledges limitations, noting that the approach still relies on the availability of intermediate helper images during training, and that automatically proposing such images for new domains remains an open question for future work.