Do vision-language models search like humans? Reasoning tokens as a reaction-time analog in classic visual-search paradigms

Technical Summary: Reasoning Tokens as a Reaction-Time Analog in Vision-Language Models

Problem Statement
Vision-language models (VLMs) have achieved fluency in describing and reasoning about images, yet it remains unclear whether their internal mechanisms for locating objects in cluttered scenes resemble human visual attention or merely produce human-like outputs via non-human processes. A century of research in human visual attention utilizes the relationship between reaction time (RT) and the number of items in a display (search slope) to diagnose underlying mechanisms: flat slopes indicate parallel, preattentive "pop-out" search, while positive slopes indicate serial, attention-demanding search. The central methodological obstacle in applying this paradigm to VLMs is the absence of a natural reaction time metric for a single model forward pass. This paper asks whether VLMs exhibit the same behavioral signatures (search slopes) as humans and whether the number of reasoning tokens generated can serve as a valid analog for search effort.

Methodology
The study adapts four classic visual-search paradigms and presents them to a suite of mid-tier and frontier VLMs (including configurations of Claude Sonnet 4.6, GPT-4o, o4-mini, Claude Opus 4.8, and GPT-5.5).

• Stimuli: Displays were programmatically rendered with varying set sizes (4, 8, 16, 32 items). The experiments included:
  1. Feature vs. Conjunction Search: Distinguishing a red 'T' among black 'T's (feature) versus a red 'T' among red 'L's and black 'T's (conjunction).
  2. Resolution Control: Blurring conjunction displays to isolate the cost of search from the cost of resolving small shapes.
  3. Spatial-Configuration Search: A 'T' vs. 'L' search without color cues.
  4. Enumeration: Counting the number of 'T's among 'L's.
  5. Search Asymmetry: Detecting a tilted bar among vertical bars versus a vertical bar among tilted bars.
• Effort Measure: Since models lack reaction time, the study uses the number of intermediate "reasoning" or "thinking" tokens emitted before the final answer as a within-model proxy for search effort. This measure is compared against a large public human benchmark (Wolfe et al., 2010) for the feature/conjunction task and against qualitative human predictions for the other tasks.
• Analysis: The study analyzes the slope of effort (tokens per item) across set sizes and correlates model effort functions with human RT functions. It also examines accuracy patterns, particularly regarding target presence/absence and enumeration errors.

Key Results
The models reproduce several human-like signatures but diverge in specific, informative ways:

• Agreements (Human Signatures):
  • Flat vs. Sloped: Feature search required flat effort (near-zero slope) across all models, while conjunction search required effort that increased with set size for models capable of deliberation.
  • Frontier vs. Mid-tier: Frontier models maintained high accuracy as set size increased, whereas mid-tier models (e.g., GPT-4o, o4-mini) saw accuracy collapse toward chance, mirroring human performance limits under high load.
  • Resolution Control: The increased effort for conjunction search was not merely due to the difficulty of resolving small shapes, as effort remained high even when images were crisp, rising only slightly at extreme blur levels.
• Divergences:
  • Target-Present/Absent Reversal: In human search, the slope for target-absent trials is typically steeper than for target-present trials (exhaustive search). VLMs reversed this: the effort slope for target-present trials was steeper, suggesting models spend more effort confirming and localizing a found target than verifying absence.
  • Enumeration: Unlike humans, who lose count and make errors as set size increases (subitizing limits), VLMs maintained near-perfect counting accuracy. They paid for this accuracy with increased computational effort (tokens) rather than accuracy loss.
  • Model-Specific Divergence (Adaptive Deliberation): A model with adaptive thinking (Claude Opus 4.8) often declined to deliberate on detection tasks, resulting in flat effort curves. Consequently, the same search asymmetry that manifested as an effort gradient in GPT-5.5 appeared as an accuracy cliff (chance-level performance) in the adaptive model.

Contributions

1. Behavioral Protocol: The paper establishes a psychophysics-style protocol for probing VLM visual search that requires no access to model internals (black-box testing).
2. Effort Metric: It validates the use of reasoning-token counts as a within-model analog for search effort, demonstrating that the shape of the effort function (slope) aligns with human reaction-time signatures despite the difference in currency (tokens vs. milliseconds).
3. Diagnostic Catalog: It provides a catalog of agreements and divergences between machine and human search, arguing that the points of divergence are as informative as the points of agreement for understanding machine visual cognition.

Significance and Claims
The paper argues that psychophysical paradigms, when applied behaviorally to VLMs, serve as a sharp and inexpensive probe of machine visual cognition. The findings suggest that frontier VLMs organize their internal computation along a difficulty axis that aligns with human attentional demand, reproducing the "fingerprint" of human visual search in the gross shape of their effort and accuracy. However, the specific mechanisms differ: models do not perform the patient, exhaustive sweeps characteristic of human target-absent search, and they trade compute for accuracy in enumeration rather than succumbing to working-memory limits. The paper concludes that the breakdown of the human-machine analogy in these specific details (absence verification, counting, and adaptive deliberation) marks the beginning of interesting questions regarding the nature of machine perception. The study remains modest, noting that token counts are an analog, not a substitute, for reaction time, and that the results are bounded by the specific models and synthetic stimuli used.