💬 NLP

CoT-Space: A Theoretical Framework for Internal Slow-Thinking via Reinforcement Learning

This paper introduces CoT-Space, a theoretical framework that models Chain-of-Thought reasoning as an optimization process within a continuous semantic space to explain the convergence of optimal reasoning length as a trade-off between underfitting and overfitting, thereby providing a principled foundation for test-time scaling via Reinforcement Learning.

Original authors: Zeyu Gan, Hao Yi, Yong Liu

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zeyu Gan, Hao Yi, Yong Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Why Don't AI Models Just Keep Thinking Forever?

Imagine you are trying to solve a tricky math problem. You could write down one step, check the answer, and stop. Or, you could write down a thousand steps, double-checking every single number, until you are absolutely sure.

In the world of AI, researchers noticed something strange. When they train Large Language Models (LLMs) to think harder (using a method called Reinforcement Learning), the models don't just keep generating longer and longer chains of thought forever. Instead, they naturally stop at a specific "sweet spot" length. If they write too little, they get the answer wrong. If they write too much, they also get it wrong (or waste time).

The paper asks: Why does the AI naturally find this perfect length and stop there?

The Problem: Counting Words vs. Understanding Thoughts

For a long time, scientists analyzed AI by looking at it word-by-word (token-by-token). They treated the AI's thinking like a train moving from one station (word) to the next.

The authors say this is like trying to understand a galaxy by counting individual stars. It's too messy and discrete. Instead, they propose looking at the galaxy itself (the "semantic space").

The Old View (Token-Level): Imagine walking through a forest where every single leaf is a separate, distinct step. It's hard to see the path.
The New View (Reasoning-Level): The authors zoom out. They say that even though the AI writes many different words to express the same idea (e.g., "calculate the total," "figure out the sum," "do the math"), those different word combinations all represent the same thought.
The Analogy: Think of the AI's reasoning space as a smooth, continuous hill. Even though the AI has to take "discrete steps" (writing specific words), the underlying landscape of ideas is smooth and continuous, like a slide.

The Core Discovery: The "Goldilocks" Trade-off

The paper argues that the AI finds the perfect reasoning length because of a classic balancing act between Underfitting and Overfitting.

1. Underfitting (Thinking Too Little)

The Analogy: Imagine trying to cross a wide river by jumping. If you only take one or two small jumps, you fall in the water.
The Paper's Claim: If the reasoning chain is too short, the AI hasn't taken enough "steps" to reach the solution. It's like trying to solve a complex math problem without doing the necessary intermediate calculations. The AI fails because it didn't think enough.

2. Overfitting (Thinking Too Much)

The Analogy: Imagine you are walking across that same river, but now you take 1,000 tiny, shaky steps. You start wobbling, getting confused by every pebble, and you might accidentally step off the path into the water because you're over-analyzing every single move.
The Paper's Claim: If the reasoning chain is too long, the AI starts to "memorize" the specific quirks of the question rather than learning the general logic. It becomes too sensitive to the exact wording of the prompt. It's like a student who memorizes the answer key for a practice test but fails the real exam because the questions are slightly different. The extra thinking introduces "noise" and confusion.

The "Noise" Metaphor: Finding the Perfect Step Size

The authors use a clever analogy from physics and machine learning involving noise (randomness).

The Setup: Imagine the AI is trying to find the bottom of a valley (the correct answer).
Too Short (Low Noise): If the AI takes huge, confident steps (very short reasoning), it might jump right over the bottom of the valley and land on the other side, missing the solution.
Too Long (High Noise): If the AI takes tiny, hesitant steps (very long reasoning), it gets lost in the details. It starts shaking and wobbling (noise) so much that it can't find the smooth path down.
The Sweet Spot: There is a "Goldilocks" amount of reasoning length that provides just the right amount of "noise" to help the AI explore the valley without getting lost or jumping over the target.

What the Experiments Showed

The researchers tested this theory with real AI models and found four key things:

Harder Problems Need Longer Paths: Just like a difficult puzzle requires more pieces, harder math problems forced the AI to naturally generate longer chains of thought to avoid "underfitting."
Smarter Models Need Shorter Paths: Surprisingly, bigger, more powerful models actually settled on shorter reasoning chains. Why? Because they are so good at learning, they don't need to over-explain. If they think too much, they start to overfit (memorize) the specific question. They need to stop earlier to stay "general" and robust.
It Doesn't Matter How You Train Them: Whether they used different training algorithms (like different types of reinforcement learning), the AI always converged to the same "optimal length" for a given problem. The length is a property of the problem and the model, not the training method.
More Noise = Shorter Paths: When the training environment was "noisier" (more random), the AI learned to take shorter, more robust paths to avoid getting confused.

The Conclusion

The paper concludes that the "convergence" of reasoning length isn't a bug; it's a feature. It is the natural result of the AI trying to balance doing enough work to solve the problem (avoiding underfitting) with not doing so much work that it gets confused or memorizes the wrong things (avoiding overfitting).

By viewing AI reasoning as a smooth, continuous journey rather than a jagged list of words, the authors show that these models are actually following the same fundamental rules of learning that humans and classical machines have followed for decades. They are just finding the perfect "Goldilocks" length to think.

Technical Summary: Why Does Reasoning Length Converge? Unveiling the Underfitting-Overfitting Trade-off in Chain-of-Thought

Problem Statement

Recent advancements in Large Language Models (LLMs), particularly those utilizing Reinforcement Learning (RL) for "test-time scaling" (e.g., DeepSeek-R1, OpenAI o1), have demonstrated that models can autonomously discover optimal Chain-of-Thought (CoT) reasoning strategies. Empirically, these models converge to a specific, optimal reasoning length ( $L_{opt}$ ) during training. However, a significant theoretical gap exists: traditional token-level analysis, which models reasoning as a discrete Markov Decision Process (MDP) over a sparse state space, fails to capture the macroscopic dynamics of reasoning-level scaling. Consequently, the phenomenon of "overthinking" (where models converge to a specific length rather than growing indefinitely) lacks a clear mechanistic explanation rooted in classical learning theory.

Methodology: The CoT-Space Framework

To bridge the gap between discrete token generation and continuous learning theory, the authors introduce CoT-Space, a novel theoretical framework that recasts the reasoning process as an optimization problem within a continuous, reasoning-level semantic space.

1. From Token-Level to Reasoning-Level

The authors argue that modeling CoT at the token level is misaligned with the nature of reasoning, where actions are abstract thoughts rather than single tokens. They propose a "zoom-out" perspective:

Semantic Equivalence: Multiple distinct token sequences can realize the same abstract reasoning step.
Continuum Approximation: As the reasoning token budget ( $K$ ) increases, the density of semantically equivalent states grows exponentially (Assumption 2.9). This allows the discrete reasoning state space to be approximated as a continuous manifold ( $\tilde{S}$ ).
Formalization: The framework defines reasoning states ( $s_o$ ), reachable minimums (successful solutions), and a reasoning loss function ( $C$ ) that quantifies the distance to the nearest optimal solution.

2. Theoretical Analysis of Convergence

Using the continuous manifold perspective, the authors analyze the convergence of CoT length through two lenses: Optimization Noise and Reasoning Risk.

Noise Perspective (Optimization Drift):
The authors model CoT generation as a Stochastic Differential Equation (SDE) on the semantic manifold. They derive a relationship where the optimization noise scale ( $\sigma$ ) is inversely proportional to the reasoning length ( $L$ ) ( $\sigma \propto 1/L$ ).
- Short CoT: High noise scale, leading to unstable trajectories that may fail to converge (Underfitting).
- Long CoT: Low noise scale, leading to trajectories that are too rigid and susceptible to overfitting specific prompt patterns.
Risk Perspective (Generalization Bound):
The authors derive an information-theoretic generalization bound for the expected population risk ( $R_{pop}$ ). The bound consists of two competing terms:
1. Optimization Drift (Underfitting Regime): Proportional to $L\sigma^2$ (or $1/L$ ). This represents the deviation from the optimal path due to excessive noise.
2. Information Complexity (Overfitting Regime): Proportional to $\sqrt{L \log L}$ . This represents the mutual information between the reasoning path and the input, which increases with length, leading to memorization of prompt-specific shortcuts and poor generalization.

The convergence to an optimal $L_{opt}$ is shown to be the natural result of minimizing the sum of these two terms, effectively balancing the trade-off between underfitting and overfitting.

Key Contributions

CoT-Space Framework: A theoretical bridge connecting discrete LLM reasoning with continuous classical learning theory by defining a reasoning-level state space that converges to a continuous manifold.
Mechanistic Explanation of $L_{opt}$ : A proof that the observed convergence of reasoning length in RL-trained models is not arbitrary but a fundamental consequence of the underfitting-overfitting trade-off.
Theoretical Predictions: The framework yields four specific, testable claims regarding the factors governing $L_{opt}$ $L_{o pt}$ :
- Task Difficulty: More complex tasks (higher smoothness parameter $\beta$ ) require longer $L_{opt}$ to reduce optimization noise.
- Model Capacity: Larger models (higher capacity) are more prone to overfitting and thus converge to shorter $L_{opt}$ to introduce necessary stochasticity.
- Algorithm Independence: The final converged $L_{opt}$ is an intrinsic property of the task and model, largely independent of the specific RL algorithm used.
- Noise Correlation: Higher optimization noise scales correlate with shorter optimal reasoning lengths.

Experimental Results

The authors empirically validate their theoretical claims using various RL algorithms (GRPO, PPO, DAPO, etc.) on mathematical reasoning benchmarks (GSM8K, MATH) with models of varying sizes (Qwen, Llama series).

Task Difficulty: Experiments confirm a monotonic increase in $L_{opt}$ as task difficulty increases (from GSM8K to MATH Level 5).
Model Capacity: Larger models consistently converge to shorter reasoning lengths compared to smaller models on the same tasks, supporting the hypothesis that larger models require less "thinking" to avoid overfitting.
Algorithm Agnosticism: Different RL algorithms guide policies to similar final $L_{opt}$ values for a given task-model pair, despite differing initial dynamics.
Noise Scale: A clear inverse correlation is observed between the optimization noise scale (proxied by KL-Loss) and the converged $L_{opt}$ .

Significance and Claims

The paper claims to provide a principled theoretical foundation for understanding internal test-time scaling in LLMs. By revitalizing foundational principles of classical machine learning (specifically noise analysis and generalization bounds), the authors argue that these theories are not obsolete but require a shift in analytical perspective (from token-level to reasoning-level) to be applicable.

The significance of this work lies in:

Explaining "Overthinking": Providing a mechanistic reason why models stop increasing reasoning length, framing it as a necessary trade-off to prevent overfitting.
Optimization Guidance: Offering a theoretical basis for optimizing reasoning trajectories, suggesting that the goal is not merely to maximize length but to find the specific $L_{opt}$ that minimizes total error.
Unifying Phenomena: The framework is presented as a conceptual tool that can potentially explain other LLM phenomena, such as hallucination (trapped in phantom minima), prompt sensitivity (landscape instability), and emergent abilities (topological phase transitions in state density), though these are presented as potential extensions of the framework rather than primary experimental validations in this work.

The authors conclude that CoT-Space offers a path toward designing more principled algorithms and building more reliable AI reasoning systems by aligning modern LLM behaviors with established learning theory.