📄 other

Humans Disengage, Reasoning Models Persist: An Item-Controlled Dissociation in Deliberation Allocation

This paper reveals a critical dissociation between humans and large reasoning models (LRMs) in deliberation allocation: while both spend more time on harder problems overall, humans invest less time on items they ultimately get wrong (abandoning difficult tasks), whereas LRMs paradoxically generate longer reasoning traces on problems they fail (persisting in uncertainty), highlighting a fundamental difference in stopping and control policies despite surface-level similarities.

Original authors: Han-yu Wang

Published 2026-07-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Han-yu Wang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching two different students take a really hard math test. One is a brilliant human teenager, and the other is a super-smart computer program designed to "think" through problems step-by-step. For a long time, scientists have been fascinated by how these two compare. They've noticed a surface-level similarity: when a problem is tricky, both the human and the computer tend to spend more time on it. It's like watching a car's speedometer; if the road gets steep, both drivers slow down to climb it. This observation led many to believe that computers are starting to "think" just like people do, using similar mental gears to figure out which problems are hard and which are easy.

But here is the twist: just because two drivers slow down on the same steep hill doesn't mean they are driving the same way. One driver might slow down to carefully navigate the turn, while the other might slow down because they are confused and staring at the map, unsure of what to do next. The big question in the world of artificial intelligence and human psychology is: how do these thinkers decide when to keep trying and when to give up? Do they spend extra effort on the problems they are struggling with, or do they cut their losses and move on? Understanding this isn't just about math scores; it's about figuring out if computers are truly learning to reason like us, or if they are just mimicking the look of thinking without the same internal logic.

The Great "Thinking" Mismatch

A new study by researcher Han-yu Wang peels back the layers of this mystery and finds a surprising secret hiding in plain sight. While humans and advanced "reasoning" AI models agree on which problems are hard, they completely disagree on what to do about it.

Think of it like a video game. When a player hits a boss that is too strong, a smart human player might realize, "This is too hard right now," and decide to stop wasting time, maybe trying a different strategy or moving on to a different level. They disengage. But the AI, according to this study, does the exact opposite. When the AI gets a problem wrong, it doesn't stop; it keeps going, generating even more text, more "thoughts," and more steps before finally giving up the wrong answer.

The Core Discovery: The "Wrong" vs. "Right" Gap
The study looked at a massive collection of data where humans and six different AI models solved the same visual puzzles (called H-ARC). The researchers measured two things:

Registration: Did they agree on which puzzles were hard? Yes. Both humans and AI took longer on the hard puzzles.
Allocation: When they got a puzzle wrong, did they spend more or less time on it compared to when they got it right? No. This is where they split.

Humans: When humans got a puzzle wrong, they spent less time on it than when they got it right. The data shows a "Cohen's d" of -0.10. This suggests a "give-up" strategy. If a human feels they are stuck, they often stop thinking about it quickly.
AI Models: When the AI got a puzzle wrong, it spent significantly more time on it. The AI models showed a "Cohen's d" ranging from 1.47 to 3.13. This means the AI kept churning out "thoughts" (tokens) even when it was clearly failing.

The paper argues that this isn't just a small glitch; it's a fundamental difference in how they "think." The human pattern looks like engagement vs. abandonment: we stay on problems we think we can solve and quit the ones we can't. The AI pattern looks like length-on-uncertainty: when the AI is unsure, it just keeps talking, generating longer chains of reasoning, even though that extra talking doesn't help it get the right answer.

The "Trace" Clues

To understand why the AI does this, the researchers looked at the actual text the AI generated (its "thought trace"). They found that when the AI was about to get a question wrong, its text was full of "hedging" and "self-doubt." It would say things like "wait, actually," "hmm," or "let me reconsider." It was essentially talking to itself, unsure of the path, but it didn't stop.

In contrast, when humans got a question wrong quickly, they often made fewer moves (like fewer grid edits in the puzzle), suggesting they had already decided the puzzle wasn't worth the effort. The study suggests that for humans, the decision to stop is a smart, metacognitive choice. For the AI, the decision to stop seems to be tied to a different rule: it keeps going until it runs out of "budget" or hits a wall, even if it knows (or should know) it's going the wrong way.

What This Means (and What It Doesn't)

The paper is very careful not to say the AI is "stupid" or that humans are "perfect." Instead, it suggests that the AI and humans are playing by different rulebooks.

The AI isn't necessarily failing to think: It might be following a rule that says, "If you are unsure, generate more text." This might be a good strategy for some tasks, but for these puzzles, it leads to wasted effort.
The "Thinking" isn't the same: Just because the AI produces a long chain of text that looks like human reasoning doesn't mean it's using the same mental stopping rules. The study shows that the AI's "thinking" is more like a hamster running on a wheel when it's confused, while the human is like a hiker who decides to turn back when the trail gets too steep.

The researchers also tested this on other types of puzzles (like intuitive physics and binary reasoning). The pattern held up: the AI kept spending extra time on its mistakes, while humans tended to spend less time. However, in one specific type of puzzle (Cortes), the AI's behavior changed slightly, showing that the "length-on-uncertainty" rule might depend on the specific game being played.

The Bottom Line

This study doesn't prove that AI can't think; it proves that AI and humans allocate their thinking differently. The surface similarity (both taking longer on hard things) hides a deep difference: humans tend to stop thinking when they realize they are stuck, while AI models tend to keep thinking (and talking) even when they are stuck.

The authors suggest that to truly understand AI, we need to look at when they stop, not just how long they think. They propose a new way to test this: if we cut off the AI's "thinking" early (truncation), we might see if those extra words were actually helping or just noise. Until we do that, the paper concludes that the AI's "persistence" on wrong answers is a sign that its internal logic for deciding "when to quit" is fundamentally different from our own. It's a reminder that in the race to build human-like AI, we might be building machines that are very good at looking like they are thinking, but very bad at knowing when to stop.

Technical Summary: Humans Disengage, Reasoning Models Persist

Problem Statement

A central question in the cognitive science of deliberation is how agents decide when to continue thinking on a problem and when to disengage. Previous research (e.g., de Varda et al., 2025) established a "cross-item" alignment between Large Reasoning Models (LRMs) and humans: both systems generate longer traces (reasoning tokens for LRMs, reaction times for humans) on items that are objectively harder. This suggests shared difficulty registration—the ability to perceive that a task is difficult.

However, this surface similarity masks a potential divergence in deliberation allocation: how an agent schedules computation after registering difficulty, specifically in relation to its own success or failure. The paper argues that cross-item correlations alone cannot distinguish between a shared mechanism of reasoning and coincidental alignment in difficulty grading. The core problem is to determine whether LRMs and humans allocate deliberation resources (time/tokens) to correct and incorrect trials in the same way when item identity is held constant.

Methodology

The study employs a two-level diagnostic framework using item-controlled, outcome-conditioned analyses on publicly available, de-identified datasets (H-ARC, INTUIT, Cortes) and a sample of six open-weight thinking LRMs (e.g., DeepSeek-R1, Qwen-QwQ, gpt-oss) plus a non-thinking baseline (DeepSeek-V3).

1. Two-Level Diagnostic

Level 1: Difficulty Registration (Cross-Item): Measures the correlation between LRM trace length and human reaction time across different items. This replicates the finding that both systems agree on which items are hard.
Level 2: Deliberation Allocation (Within-Item): Measures the relationship between deliberation length and correctness within the same agent and within the same item. This is the novel contribution. It asks: "Does the agent spend more deliberation on trials it gets right or wrong?"

2. Statistical Approach

Metric: The primary metric is the d-ratio (Cohen's d on log-deliberation), calculated as $(\bar{\ell}_{wrong} - \bar{\ell}_{right}) / s_{pooled}$ $(\overset{ˉ}{ℓ}_{w r o n g} - \overset{ˉ}{ℓ}_{r i g h t}) / s_{p oo l e d}$ .
- Positive d: Agent deliberates longer on failures.
- Negative d: Agent deliberates longer on successes.
Controls: To isolate allocation from difficulty, the authors use item fixed-effects regressions. This absorbs item identity, ensuring the comparison is driven purely by the correctness outcome within the same item.
Scope: The analysis is "item-controlled" rather than strictly "within-agent" in a repeated-measures sense, as the public data provides one outcome per (item $\times$ model) pair. The contrast is identified across models (and humans) attempting the same specific item.

3. Mechanistic Probes

To interpret the behavioral divergence, the authors analyzed trace content:

Human Data: Decomposed reaction time using "action counts" (grid edits) to distinguish between "abandonment" (fast errors) and "engaged failure" (slow errors).
LRM Data: Analyzed reasoning traces for "self-doubt markers" (hedging), self-correction markers, and lexical repetition rates to see if longer traces correlate with uncertainty.

Key Results

1. The Allocation Gap (Sign Reversal)

On the primary dataset (H-ARC), a stark dissociation emerges:

Humans: Exhibit a negative d-ratio ( $d = -0.10$ ). When controlling for item difficulty, humans spend less time on trials they get wrong. This suggests a policy of engagement vs. abandonment: humans persist on items they expect to solve and disengage quickly on those they do not.
Thinking LRMs: Exhibit a positive d-ratio ( $d = 1.47–3.13$ ). When controlling for item difficulty, LRMs spend more tokens on trials they get wrong. This suggests a policy of length-on-uncertainty: the model generates longer chains when it is unsure, which is precisely when it tends to fail.

2. Robustness and Generalization

Cross-Paradigm: The sign reversal holds on INTUIT (intuitive physics), where LRMs show $d > 0$ and humans show $d \approx 0$ .
Boundary Case (Cortes): On Cortes (binary relational reasoning), the LRM-only within-item slope becomes positive (opposite to H-ARC), but the interaction between agent type and correctness remains significant because humans show an even stronger positive slope. The dissociation survives, though the mechanism reading is more complex here.
Baseline: The non-thinking DeepSeek-V3 also shows a tendency to spend more tokens on wrong trials (positive d-ratio), suggesting this "length-on-uncertainty" behavior may be a property of base-model generation under uncertainty, not just a "reasoning" feature.

3. Mechanistic Evidence

Human Engagement: The positive within-item slope for humans (on correct trials) is mediated by action counts. Fast-wrong trials (abandonment) have significantly fewer actions than slow-wrong trials (engaged failure), supporting the "engagement vs. abandonment" hypothesis.
LRM Uncertainty: On H-ARC, wrong-trial chains contain significantly more self-doubt/hedging markers. On Cortes, they show higher 5-gram repetition and lower type-token ratios. These surface markers suggest the model is "stuck" or uncertain, generating more tokens without necessarily improving accuracy.

Key Contributions

Decoupling Registration and Allocation: The paper separates two behavioral levels often conflated in human-model comparisons. It demonstrates that while LRMs and humans align on which items are hard (registration), they diverge fundamentally on how they allocate resources to those items (allocation).
Item-Controlled Diagnostic: It introduces a rigorous, outcome-conditioned diagnostic that holds item identity fixed. This reveals that cross-item correlations (difficulty registration) are orthogonal to within-item deliberation policies.
Reframing Alignment: The paper argues that apparent alignment in trace length is a "surface similarity" that hides opposite internal policies. It reframes human-LRM alignment not as a question of trace length alone, but as a question of stopping and control policies.

Significance and Claims

The paper claims that the observed dissociation challenges the notion that current LRMs implement human-like metacognitive control.

Metacognitive Decoupling: In humans, monitoring (uncertainty) is coupled with control (stopping/abandoning). In LRMs, monitoring signatures (hedging, uncertainty markers) rise on wrong trials, but they are weakly coupled to stopping; the model continues to generate tokens even when the outcome is likely to be incorrect.
Resource-Rational Interpretation: The human pattern is consistent with a resource-rational policy where computation is withdrawn when expected marginal gain vanishes (abandonment). The LRM pattern suggests a policy where extra tokens are allocated to uncertain items regardless of their eventual utility, potentially optimizing for a different objective (e.g., training signals, verifier rewards) rather than item accuracy.
Methodological Shift: The authors assert that future work comparing LRMs to human cognition must pair cross-item alignment tests with within-item allocation tests. Relying solely on cross-item correlations is insufficient to claim shared reasoning mechanisms.

Limitations: The authors explicitly note that the data are observational. They do not prove that the extra tokens on LRM wrong trials have zero causal value, only that their allocation pattern is the inverse of human behavior. They propose a "controlled truncation" experiment as the necessary next step to determine if the extra tokens are "useful search" or "padding." The study is limited to open-weight models and does not test closed-frontier systems.