Technical Summary: MENTISOCULI: Revealing the Limits of Reasoning with Mental Imagery

Problem Statement

Current frontier multimodal models are transitioning from Multimodal Large Language Models (MLLMs), which treat vision as a passive input, to Unified Multimodal Models (UMMs) capable of native, interleaved generation of text and images. This shift has sparked interest in using intermediate visualizations as a reasoning aid, analogous to human "mental imagery." However, it remains unclear whether models can effectively form, maintain, and manipulate visual representations to solve complex reasoning tasks, or if such visual thoughts actually improve performance over text-only reasoning. Existing benchmarks often evaluate reasoning about visual information rather than reasoning with mental imagery, or they lack the procedural rigor to disentangle generation errors from reasoning deficits.

Methodology

To address these gaps, the authors introduce MENTISOCULI (Latin for "eyes of the mind"), a procedural, stratified benchmark designed to challenge models' ability to reason with visual representations.

Benchmark Design

MENTISOCULI consists of five multi-step visual reasoning tasks, each procedurally generated across five difficulty levels (Level 1 to Level 5). The tasks are designed to be difficult to textualize but intuitive for humans to solve visually, adhering to six key desiderata:

1. Visual Nature: Tasks rely on spatial relations, geometric constraints, and object transformations rather than common knowledge.
2. High Information Density: Tasks avoid simple grid worlds, utilizing complex shapes and continuous transformations.
3. Sequential Manipulation: Tasks require repeated updates to a mental state, where actions depend on previous outcomes.
4. Procedural: Tasks are algorithmically generated with ground-truth visual chains of thought (CoT) for analysis.
5. Stratified: Difficulty is controlled by the number of steps or objects, allowing for the identification of model breaking points.
6. Generative Feasibility: Tasks are constrained to 2D projections with legible details at standard resolutions.

The five tasks are:

• Form Board: Selecting a subset of shapes to fill a target silhouette without gaps or overlaps.
• Hinge Folding: Predicting discrete rotation angles for a chain of polygons to match a target silhouette.
• Paper Fold: Identifying the correct unfolded hole pattern after a sequence of folds and a hole punch.
• Rush Hour: Navigating a red vehicle out of a crowded lot by moving blocking vehicles (with continuous positions and non-axis-aligned orientations).
• Sliding Puzzle: Restoring a permuted natural image by moving an empty tile.

Evaluation Framework

The authors evaluated a spectrum of model families:

• MLLMs: Text-only outputs (e.g., Gemini 3, GPT-5.1, Qwen3-VL).
• Latent Reasoning Models: Fine-tuned models generating interleaved visual latent tokens (e.g., Mirage, LatentSketchpad).
• Unified Multimodal Models (UMMs): Models explicitly generating images interleaved with text (e.g., Gemini 2.5-I, Gemini 3-I, Emu 3.5).
• Video Models: Purely visual rollouts conditioned on a prompt (e.g., Veo 3.1, Wan 2.6).

Evaluation involved automated scoring of text outputs (via simulation or exact matching) and a custom video rater for video models. The study also included a psychophysical experiment with human participants (PhD students) to establish a performance ceiling and analyze reasoning effort allocation.

Key Contributions

1. MENTISOCULI Benchmark: A rigorous, procedural benchmark for multi-step reasoning with mental imagery, featuring ground-truth visual CoTs and stratified difficulty levels.
2. Comprehensive Analysis: An evaluation of the full spectrum of machine mental imagery, from implicit latent tokens to explicit generated images and video.
3. Diagnosis of Failure Modes: Evidence that UMM failures stem not only from generation errors but also from a critical inability to interpret and leverage visual aids (even ground-truth ones) for decision-making.
4. Human-Machine Comparison: Data highlighting that models fail to adaptively increase reasoning effort (token usage) in response to increasing visual-spatial complexity, unlike humans.

Results

1. Ineffectiveness of Explicit Visual Thought

Across all tasks and difficulty levels, explicit visual strategies (latent tokens, interleaved images, or video) did not improve performance over text-only baselines.

• MLLMs vs. UMMs: UMMs (e.g., Gemini 3-I) frequently performed worse than their text-only counterparts (e.g., Gemini 3).
• Latent Reasoning: Fine-tuned latent reasoning models showed marginal gains on easier levels but collapsed to near-chance performance at Level 5, offering no consistent advantage over text-only fine-tuning.
• Video Models: Video models (Veo 3.1, Wan 2.6) failed to exceed chance performance, breaking down quickly as difficulty increased.

2. The Dual Failure of UMMs

Analysis of UMMs revealed two compounding issues:

• Generation Errors: Models frequently hallucinate state changes, losing object identity or geometry across frames.
• Interpretation Errors: Crucially, even when provided with oracle visualizations (ground-truth intermediate images), UMMs failed to leverage them to improve reasoning. In many cases, models with oracle visuals performed no better than those generating their own (flawed) images, indicating an inability to treat visual states as actionable evidence for decision-making.

3. Decoupling of Modalities

In UMMs, the text reasoning channel and the image generation channel are weakly coupled. On the Rush Hour task, the two channels often solved different puzzles. As difficulty increased, the text channel carried the planning burden while the image channel failed, yet the model did not successfully integrate the two.

4. Limits of Reasoning Enhancements

Standard techniques that improve language-based reasoning—In-Context Learning (ICL), Prompt Optimization, Increased Reasoning Budget, and Tool Use—yielded no systematic gains for visual reasoning tasks. Increasing token usage did not correlate with improved accuracy on harder levels.

5. Human vs. Machine Effort

Humans demonstrated adaptive reasoning, spending more time (and by proxy, effort) on harder puzzles. In contrast, models like Gemini 3 did not increase token usage as task difficulty increased from Level 3 to Level 5, suggesting a lack of adaptive reasoning depth.

Significance and Claims

The paper concludes that despite the intuitive appeal of "mental imagery" for reasoning, current frontier models cannot yet bridge the gap between generation and reasoning. The authors assert that:

• Visual thoughts do not currently benefit model reasoning; in fact, they often introduce compounding errors.
• The failure is not merely a lack of generative capability but a fundamental inability to maintain consistency and utilize visual information for planning.
• MENTISOCULI provides the necessary foundation to isolate these failure modes and distinguish reasoning capacity from representational alignment.

The work suggests that achieving effective machine mental imagery will likely require dedicated training data and a shift in focus by model developers toward multi-step visual reasoning, rather than simply scaling current architectures. The paper remains modest, noting that while visual aids can be helpful in specific settings, their effectiveness is neither uniform nor reliable with current technology.