Humans Disengage, Reasoning Models Persist: Separating Difficulty Registration from Deliberation Allocation

Technical Summary: Humans Disengage, Reasoning Models Persist

Problem Statement
Recent work (de Varda et al., 2025) established a surface-level alignment between Large Reasoning Models (LRMs) and humans: both systems generate longer reasoning traces (or take more time) on harder items. This "cross-item alignment" suggests that LRMs register difficulty similarly to humans. However, this correlation conflates two distinct cognitive processes: difficulty registration (grading items by difficulty) and deliberation allocation (deciding how much computation to spend on a specific item once its difficulty is known).

The paper argues that cross-item correlations are insufficient to claim mechanistic similarity because they do not reveal how an agent schedules computation relative to its own successes and failures. Specifically, it is unknown whether LRMs, like humans, allocate more effort to items they expect to solve (engagement) or whether they allocate effort based on a different signal, such as uncertainty, even when that effort leads to failure.

Methodology
The study introduces a two-level diagnostic to separate registration from allocation, utilizing a public, matched human–LRM corpus across three reasoning paradigms: H-ARC (visual abstraction), INTUIT (intuitive physics), and Cortes (binary relational reasoning).

1. Difficulty Registration (Probe 1): The authors replicate the cross-item analysis, correlating LRM reasoning trace length with human reaction time (RT) across items.
2. Deliberation Allocation (Probe 2): The core innovation is an outcome-conditioned, item-controlled diagnostic. The authors ask: Given a fixed item, does the agent spend more deliberation on trials it answers correctly versus incorrectly?
   • Metric: The within-agent $d$-ratio, defined as the Cohen's $d$ on log-deliberation time (seconds for humans, reasoning tokens for LRMs) between wrong and right trials: $d = (\bar{\ell}_{wrong} - \bar{\ell}_{right}) / s_{pooled}$.
   • Controls: To isolate allocation from difficulty, the analysis uses item fixed effects (absorbing item identity) and difficulty-controlled regressions. This ensures the comparison is strictly within the same item, preventing cross-item difficulty gradients from confounding the result.
   • Normalization: Deliberation is measured within each agent's native scale (seconds vs. tokens); absolute magnitudes are never compared directly.
   • Models: Six open-weight thinking LRMs (e.g., DeepSeek-R1, Qwen-QwQ, gpt-oss) and one non-thinking baseline (DeepSeek-V3) are compared against matched human samples.

Key Results
The study finds a dissociation between registration and allocation:

• Registration (Alignment): Thinking LRMs and humans agree on which items are hard. LRM trace length positively correlates with human RT across items (Spearman $\rho = 0.16–0.30$), replicating de Varda et al. (2025).
• Allocation (Divergence): Once item identity is fixed, the two systems exhibit opposite allocation policies:
  • Humans: Show a negative or near-zero $d$-ratio ($d = -0.10$ on H-ARC). Humans spend less time on trials they get wrong, consistent with an engagement-vs-abandonment strategy: they persist on items they expect to solve and disengage (abandon) on those they do not.
  • Thinking LRMs: Show a large positive $d$-ratio ($d = 1.47–3.13$ on H-ARC). LRMs spend more tokens on trials they get wrong. This suggests a length-on-uncertainty policy: chains grow when the model is unsure, which is precisely when it tends to fail.
  • Non-Thinking Baseline: The non-thinking DeepSeek-V3 also shows a positive $d$-ratio on some tasks, suggesting that length expansion under uncertainty may be a property of base generation rather than a specific "reasoning" allocation policy, though the thinking models show stronger effects.
• Trace Content: Inside the LRM traces, wrong trials contain higher densities of self-doubt markers (e.g., "wait," "hmm") on H-ARC and higher lexical repetition on Cortes, supporting the interpretation that length is driven by uncertainty rather than productive search.

Contributions

1. Conceptual Decomposition: The paper formally separates difficulty registration from deliberation allocation, demonstrating that cross-item alignment does not imply shared stopping policies.
2. New Diagnostic: It introduces the outcome-conditioned, item-controlled $d$-ratio as a necessary test for human–LRM alignment in metacognitive control.
3. Empirical Finding: It reveals that while LRMs and humans agree on what is hard, they disagree fundamentally on how to respond to that difficulty. The apparent alignment in prior work masked an opposite allocation policy.

Significance and Claims
The paper claims that the observed dissociation challenges the view that LRMs implement human-like metacognitive control.

• Metacognitive Decoupling: The results suggest that in current thinking LRMs, uncertainty monitoring (signaled by hedging or length) is decoupled from stopping control. Humans use monitoring to gate effort (abandoning low-value trials), whereas LRMs appear to continue generating tokens when uncertain, regardless of the diminishing return on accuracy.
• Resource-Rational Interpretation: Under a resource-rational framework, the human pattern is consistent with a stopping rule that withdraws effort when expected marginal gain vanishes. The LRM pattern implies either a failure to condition expected gain on item-specific features or an optimization objective different from task accuracy (e.g., length-conditioned preferences).
• Methodological Shift: The authors argue that future cognitive modeling of LRMs must pair cross-item registration tests with within-item allocation tests. Relying solely on cross-item correlations is insufficient to claim mechanistic similarity.

Limitations and Future Directions
The authors explicitly state that their findings are observational and do not prove the causal value of the extra tokens. They do not claim that LRMs are "irrational," but rather that they implement a different resource-rational policy. The paper proposes controlled inference-time truncation as the necessary next step to discriminate between "length-on-uncertainty" (where truncation would not harm accuracy) and "useful search" (where truncation would degrade performance), a test not performed in this study. The scope is limited to open-weight models and specific datasets, and the engagement-vs-abandonment reading is an interpretation of behavioral signatures, not a direct observation of internal architecture.