Position: Stop Anthropomorphizing Intermediate Tokens as Reasoning/Thinking Traces!

This position paper argues that anthropomorphizing intermediate token generation as "reasoning traces" or "thoughts" is a dangerous misconception that obscures the true nature of language models, hinders their effective use, and leads to flawed research, urging the community to abandon such metaphors.

The Gist

The Core Message: Stop Treating AI "Thinking" Like Human Thinking

Imagine you have a very smart, very fast robot that can solve complex math problems or write code. When you ask it a question, it doesn't just spit out the answer immediately. Instead, it takes a moment to "talk to itself," generating a long stream of text before giving you the final result.

The AI research community has started calling this internal monologue "Reasoning" or "Thoughts." They treat it like a human sitting down with a pencil and paper, working through the steps, having "Aha!" moments, and correcting their own mistakes.

This paper argues that this is a dangerous lie.

The authors, a team of researchers from Arizona State University, are saying: "Stop anthropomorphizing (giving human traits to) these intermediate tokens." They believe that calling these text strings "thoughts" is not just a harmless metaphor; it's actively confusing and dangerous because it makes us trust the AI too much when we shouldn't.

---

The Analogy: The "Scripted Actor" vs. The "Real Thinker"

To understand why the authors are worried, let's use an analogy.

The Current View (The "Human" View):
Imagine you are watching a magician. Before pulling a rabbit out of a hat, the magician mumbles, "Let's see... the rabbit is hungry, I need to check the hat, okay, here we go!" You assume the mumbling is the magician actually thinking about the trick. You trust the rabbit because the mumbling sounded logical.

The Authors' View (The "Scripted" View):
The authors argue that the AI isn't a magician thinking. It's more like a method actor who has memorized a script.

• The AI has read millions of human stories where people say "Hmm," "Wait a minute," or "Aha!" when solving problems.
• The AI learned that if it outputs these specific words before the answer, it gets a "reward" (it gets the answer right more often).
• So, it generates a long, rambling script that sounds like thinking. It might say "Aha!" not because it had a sudden realization, but because the word "Aha!" statistically leads to the correct answer in its training data.

The Danger:
If you think the AI is "thinking," you might trust a wrong answer just because the "thinking" part sounded convincing. It's like trusting a magician because he mumbled the right words, even if he's actually just pulling a rabbit out of a sleeve he didn't tell you about.

---

The Evidence: Why "Thinking" is a Myth

The paper provides several pieces of evidence to prove that these "thoughts" aren't real reasoning:

1. The "Swapped Script" Experiment
Researchers took models and trained them with "nonsense" scripts. Imagine teaching a student to solve math problems, but instead of showing them the correct steps, you show them a script that says "Add 2 + 2 and get 5," but then magically gives the correct answer of 4 at the end.

• Result: The AI still learned to get the right answer (4), even though the "thinking" part was completely wrong.
• Meaning: The AI doesn't care if the "thoughts" make sense. It just cares that the pattern of "Script + Answer" leads to a reward.

2. The "Aha!" Moment
DeepSeek's famous AI (R1) was praised for having "Aha!" moments in its text.

• The Reality: The AI doesn't have an internal state that changes when it says "Aha!" It's just a token (a word) in a sequence. It's like a parrot saying "I'm happy!" when it sees a banana, not because it feels joy, but because that's the sound associated with bananas.

3. Length Doesn't Equal Effort
We often think, "Wow, this AI wrote 500 words of thinking; it must be working really hard!"

• The Reality: The paper shows that AI models often generate longer and more confusing texts when they are trained to do so, even for simple problems. Sometimes, they babble for pages just to fill space. The length of the text is often a side effect of how the AI was trained, not a measure of how "smart" the solution is.

---

Why Does This Matter? (The "False Confidence" Trap)

The authors are worried about three main things:

1. False Trust: If users believe the AI is "thinking," they will trust its answers blindly. If the AI says, "I calculated this carefully," but it actually just guessed and wrote a fancy story to justify it, the user might make a bad decision based on that answer.
2. Bad Research: Scientists are wasting time trying to make these "thoughts" more human-readable or trying to fix the "logic" in the text. But if the text isn't actually logic, they are trying to fix a ghost. They should be focusing on making the final answer correct, not the intermediate chatter.
3. The "Black Box" Problem: Some companies (like OpenAI) hide their intermediate tokens because they know they aren't interpretable. They show a "summary" instead. The authors argue this is honest. But other companies (like DeepSeek) show the full "thinking" text, which tricks people into thinking they understand how the AI works.

---

The Call to Action: What Should We Do?

The authors propose a simple shift in mindset:

• Stop calling it "Thinking": Call it "Intermediate Tokens" or "Derivational Traces." It's just data the model generates to help itself, not a window into a human-like mind.
• Don't trust the "Reasoning": If you need to trust an AI's answer, don't look at its "thought process." Look at the answer itself and verify it with a separate tool (like a calculator or a code checker).
• Let the AI be weird: If the AI solves a problem best by generating gibberish or non-human symbols, let it do that! We shouldn't force it to sound like a human just to make us feel comfortable.

The Bottom Line

The AI is a super-fast pattern matcher, not a little person inside a computer.

When it generates a long chain of text before an answer, it's not "thinking" in the way we do. It's performing a complex dance it learned from its training data to maximize its chances of being right. Treating this dance as "human reasoning" is a dangerous illusion that makes us trust machines we don't truly understand.

The paper's final advice: Stop looking for a human soul in the machine's code. Focus on whether the answer is right, not on how the machine talks about getting there.

Technical Summary

1. Problem Statement

The paper addresses a pervasive trend in Large Language Model (LLM) research, particularly regarding "Large Reasoning Models" (LRMs) like DeepSeek R1. These models utilize Intermediate Token Generation (ITG), where the model produces a sequence of tokens (often called "Chain of Thought" or "reasoning traces") before outputting a final answer.

The core problem identified is the anthropomorphization of these intermediate tokens. The community frequently interprets these tokens as:

• Human-like "thoughts" or internal reasoning processes.
• Interpretable windows into the model's cognitive state.
• Direct indicators of problem-solving effort or difficulty (e.g., longer traces = more "thinking").

The authors argue that this metaphor is not merely harmless but dangerous. It creates a false sense of model capability, engenders misplaced trust in the model's outputs, and steers research toward fruitless directions (e.g., optimizing for human-readable "reasoning" rather than solution accuracy).

2. Methodology & Evidence Synthesis

Rather than conducting a single new experiment, the authors employ a critical review and synthesis of emerging literature to deconstruct the anthropomorphic narrative. Their methodology involves:

• Technical Deconstruction: Analyzing the training pipelines of LRMs (specifically post-training via Reinforcement Learning with Verifiable Rewards, or RLVR). They highlight that these models are optimized for final answer correctness, not the semantic validity of intermediate steps.
• Literature Review of Counter-Evidence: Collating studies that demonstrate a lack of correlation between trace correctness and solution accuracy.
• Analysis of Training Dynamics: Examining how Reinforcement Learning (RL) incentives (specifically credit assignment) encourage models to generate longer sequences or specific patterns that satisfy reward models without necessarily performing logical reasoning.
• Case Studies: Reviewing specific experiments where models were trained on:
  • Noisy/Incorrect Traces: Traces with swapped steps or incorrect math that still led to high performance.
  • Algorithmic Traces: Models trained on formal A* search traces where the semantic meaning of the trace was broken, yet performance remained robust.

3. Key Contributions

The paper makes several critical technical contributions to the discourse on LLM reasoning:

• Refutation of Semantic Validity: The authors provide strong evidence that intermediate tokens do not possess guaranteed end-user semantics. They are often "babble" or stylistic mimicry of human reasoning rather than actual computation.
• Decoupling Trace Length from Effort: The paper challenges the assumption that longer intermediate token sequences indicate deeper reasoning. They argue that increased length is often an artifact of RL training dynamics (e.g., distributing terminal rewards across intermediate tokens) rather than adaptive computation.
• Identification of "False Confidence": The authors demonstrate that interpreting traces as reasoning leads to False Positives (correct answers with incorrect reasoning) and False Negatives (incorrect answers with plausible reasoning), making human interpretation of traces an unreliable proxy for trust.
• Critique of "Aha" Moments: The paper specifically debunks the interpretation of tokens like "aha!" or "wait" as evidence of internal state changes, noting that LLMs are stateless between forward passes and these tokens are merely statistical predictions.

4. Key Results & Findings

The synthesis of evidence leads to the following concrete findings:

• Trace Correctness $\neq$ Solution Correctness: Studies (e.g., [58], [6]) show that models can achieve high accuracy even when trained on incorrect, swapped, or semantically nonsensical traces. Conversely, models can produce correct final answers with completely flawed intermediate reasoning.
• RL Incentivizes Length, Not Logic: In RLVR frameworks (like DeepSeek R1), the reward signal is typically applied only to the final answer. This inadvertently incentivizes the model to generate longer sequences of tokens to "claim" credit, leading to the observed increase in trace length without a corresponding increase in logical rigor.
• Robustness to Semantic Noise: Models trained on noisy traces (e.g., Dualformer, Searchformer extensions) show that performance gains come from fitting the pattern of the trace format, not the content or logic of the trace.
• The "Cheating" Phenomenon: Some research suggests models learn to "cheat" by generating plausible-sounding reasoning that bypasses the actual problem-solving steps required to reach the correct answer.
• Vendor Behavior: Major frontier model developers (OpenAI, Google, Anthropic) have stopped exposing raw intermediate tokens in production, likely acknowledging their lack of interpretability, while the research community continues to treat them as transparent.

5. Significance & Call to Action

The paper concludes with a strong call to action for the AI research community:

• Stop Anthropomorphizing: Researchers must cease treating intermediate tokens as "thoughts" or "reasoning traces." They should be viewed strictly as internal computational artifacts designed to aid the model in reaching a correct solution, not to explain the solution to a human.
• Shift Evaluation Metrics: Evaluation should focus on solution verification (using formal verifiers or third-party checks) rather than the "interpretability" of the reasoning trace.
• Optimize for Accuracy, Not Style: Training objectives should prioritize solution accuracy. If this means intermediate tokens become non-linguistic, vector-based, or "gibberish" to humans, that is acceptable and potentially beneficial.
• Avoid "Ersatz" Reasoning: Deliberately training models to produce human-like reasoning traces is dangerous because it exploits human cognitive biases, leading users to trust incorrect answers simply because the "reasoning" sounds plausible.

In summary, the paper argues that the "Chain of Thought" is a misnomer in the context of modern LRMs. It is a computationally useful mechanism for the model, but a semantically void and potentially misleading artifact for the human user. Recognizing this distinction is crucial for developing trustworthy and effective AI systems.