Test-time Scaling over Perception: Resolving the Grounding Paradox in Thinking with Images

1. Problem Definition: The Grounding Paradox

The paper identifies a fundamental limitation in current Multimodal Large Language Models (MLLMs) that utilize "Thinking with Images" (i.e., using tools like zooming and cropping to inspect image regions).

• The Core Issue: Existing systems suffer from the Grounding Paradox. To answer a complex visual query, a model must invoke a visual tool (e.g., zoom) to inspect a specific region $R^*$. However, selecting the correct region $R^*$ requires fine-grained discriminative information that is only available after the tool has been invoked.
• The Consequence: Models are forced to make high-stakes decisions (where to look) based on coarse, global perception that lacks the necessary detail. This leads to a circular dependency where models frequently inspect irrelevant regions, miss critical evidence, or fail to invoke tools when needed, resulting in brittle reasoning in fine-grained tasks (e.g., small text recognition, dense charts).
• Limitation of Current Approaches: Simply improving tool-call formatting or training on specific distributions does not solve the underlying inference problem: the inability to reliably localize evidence under uncertainty.

2. Methodology: Test-time Scaling over Perception (TTSP)

The authors propose TTSP, a framework that treats perception itself as a scalable inference process. Instead of relying on a single, potentially misgrounded trajectory, TTSP allocates test-time compute to explore, validate, and refine visual evidence iteratively.

The framework operates in $N$ rounds, each consisting of three coupled steps:

A. Parallel Perceptual Exploration

In each round, TTSP generates $K$ parallel perception traces. To balance exploration and exploitation, it divides these into two pools:

• Fresh Traces ($\alpha$ fraction): Generated without access to prior knowledge to ensure broad coverage of the visual space and prevent confirmation bias.
• Guided Traces ($1-\alpha$ fraction): Conditioned on structured knowledge from previous rounds to focus on unresolved conflicts and build upon validated facts.

B. Reliability Scoring and Filtering

Not all traces are reliable. TTSP employs a lightweight, entropy-based filtering mechanism:

• Entropy Estimation: It calculates the Shannon entropy of the model's output distribution at token positions, specifically focusing on high-uncertainty decision points (e.g., selecting bounding boxes or final answers).
• Scoring: A reliability score is computed as the negative mean entropy of the top-$k'$ highest-entropy tokens.
• Filtering: The bottom $\rho$ fraction of traces (lowest reliability) are discarded to prevent noisy or hallucinated observations from contaminating the knowledge base.

C. Structured Knowledge Extraction and Iterative Refinement

The surviving traces are distilled into a Structured Knowledge Memory consisting of two tiers:

1. Confirmed Knowledge: Observations consistently supported across traces and verified against the image. These are treated as facts to reduce redundant re-inspection.
2. Open Conflicts: Unresolved disagreements between traces. Each conflict includes competing claims and a spatial directive indicating which region to inspect in the next round.

This memory is injected into the next round's system prompt. Guided traces use "Confirmed Knowledge" as a foundation and target "Open Conflicts" for resolution. This transforms tool use from one-shot guesswork into an iterative process of perceptual hypothesis testing.

D. Final Aggregation

After the final round, the answer is derived via reliability-weighted aggregation. Each retained trace casts a vote for its predicted answer, weighted by its reliability score ($w_i = \exp(\text{score}(\tau_i)/\gamma)$), allowing the system to leverage collective evidence while prioritizing high-confidence trajectories.

3. Key Contributions

1. Identification of the Grounding Paradox: The paper formally defines the circular dependency between perception and action in tool-augmented MLLMs, explaining why current "Thinking with Images" systems fail systematically.
2. TTSP Framework: A novel test-time scaling method that extends scaling from reasoning (text) to active perception. It introduces a mechanism for multi-trajectory exploration, confidence-aware filtering, and iterative knowledge refinement.
3. Empirical Validation: Comprehensive experiments demonstrating that TTSP outperforms strong baselines (including GPT-4o, DeepEyes, and concurrent methods like RTWI) across diverse benchmarks.

4. Experimental Results

The authors evaluated TTSP on high-resolution benchmarks (V*Bench, HR-Bench at 4K/8K, TreeBench) and general reasoning benchmarks (MME-RealWorld-Lite).

• Performance: TTSP consistently achieved state-of-the-art results across all benchmarks and backbone sizes (Qwen3-VL-4B and 8B).
  • On TreeBench, TTSP improved overall performance by significant margins (e.g., +1.5% over the next best method on the 8B backbone).
  • It showed particularly strong gains in tasks requiring fine-grained perception and precise evidence localization.
• Scalability:
  • Model Size: TTSP scales favorably with larger backbones, suggesting it complements parameter scaling rather than replacing it.
  • Width & Depth: Performance improved with increased perception width (more traces per round) and depth (more iterative rounds), confirming that both parallel exploration and sequential refinement are beneficial.
• Efficiency: Despite using multiple rounds, TTSP was more token-efficient than the concurrent method RTWI. By consolidating shared observations into structured knowledge, it avoided redundant re-inspection of the same regions.
• Ablation Studies: Removing any component (Reliability Filtering, Structured Knowledge, or Weighted Aggregation) led to performance degradation, proving that the synergy of these components is essential.

5. Significance and Conclusion

The paper argues that the primary bottleneck in multimodal reasoning is not the ability to reason over observed facts, but the reliable discovery of those facts under perceptual uncertainty.

• Paradigm Shift: TTSP shifts the focus from "scaling reasoning" to "scaling perception." It demonstrates that treating perception as an iterative, scalable inference process is a viable path to robust multimodal reasoning.
• Generalizability: The method is model-agnostic (works with different backbones) and addresses a fundamental structural issue in tool-augmented agents, offering a principled solution to the Grounding Paradox.
• Future Impact: This work suggests that future advancements in MLLMs for complex visual tasks will depend less on simply making models larger and more on developing better mechanisms for active, uncertainty-aware evidence acquisition at test time.