🤖 machine learning

Uncovering the Representation Geometry of Minimal Cores in Overcomplete Reasoning Traces

This paper introduces the concept of "minimal cores" to demonstrate that language model reasoning traces are significantly overcomplete, revealing that a small subset of steps preserves predictive accuracy while offering a cleaner, lower-dimensional geometric representation of the underlying reasoning process.

Original authors: Sanjoy Chowdhury, Dinesh Manocha

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sanjoy Chowdhury, Dinesh Manocha

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you ask a very smart, but slightly chatty, friend to solve a math problem or answer a tricky question. They don't just give you the answer; they write out a long, detailed story of how they got there. They might rephrase the question, check their work three times, offer alternative ways to solve it, and add little notes about why they chose a certain path.

This paper asks a simple question: How much of that long story is actually necessary to get the right answer?

The authors call these long stories "overcomplete reasoning traces." They found that, on average, nearly half of the steps in these stories are actually just "fluff"—redundant checks, extra explanations, or alternative routes that the model doesn't need to keep its final answer correct.

Here is a breakdown of their findings using some everyday analogies:

1. The "Core" vs. The "Fluff"

Think of a reasoning trace like a long, winding road trip to a destination.

The Full Trace: This is the entire drive, including every scenic detour, every time you stopped to check the map, every time you took a wrong turn and corrected it, and every time you explained to your passenger why you were taking that route.
The Minimal Core: The authors developed a method to strip away the detours and find the shortest, most direct highway that still gets you to the exact same destination.

They found that for many problems, you can remove about 46% of the steps (the scenic detours and map checks) and the model will still give you the exact same answer.

2. The "Star Players"

If you look at a sports team, you might think every player on the field is equally important. But this paper suggests that in a reasoning trace, a few steps do almost all the heavy lifting.

They found that the top 3 steps in a reasoning chain are responsible for about 65% of the "necessity."
Imagine a play in a movie where the hero saves the day. The script might have 10 lines of dialogue, but only 3 lines are actually the "save." The other 7 lines are just setting the scene or building tension. The model's "answer" relies heavily on those few critical lines.

3. Cleaning Up the "Messy Room"

The authors looked at the "geometry" of these thoughts, which is a fancy way of looking at the shape and structure of the data.

Full Traces: Think of these as a cluttered room. There's everything in there: the useful tools, the trash, the decorations, and the furniture. It's hard to find the important stuff because it's mixed with the noise.
Minimal Cores: When they extract the minimal core, it's like cleaning the room. You throw out the trash and the decorations, leaving only the essential tools.
The Result: The "cleaned" room (the minimal core) is not just smaller; it's clearer. The authors found that when they looked at these cleaned-up traces, it was much easier to tell the difference between a "correct" thought process and an "incorrect" one. The messy room made it hard to see the truth; the clean room made the truth stand out.

4. The "Universal Translator"

One of the coolest findings is that these "minimal cores" work across different AI models.

Imagine you have a map drawn by Model A. Usually, if you give that map to Model B, it might get confused because the styles are different.
But when you strip Model A's map down to just the essential "minimal core" (the direct highway), Model B can use it perfectly.
This suggests that the "core" isn't just specific to one model's style; it captures the actual logic of the problem in a way that other models can understand, even if they were trained differently.

What They Did (The Method)

To find these cores, they didn't just guess. They used a "greedy backward elimination" strategy.

Imagine you have a long chain of paperclips holding a heavy weight.
They start at the end and ask: "If I remove this specific paperclip, does the weight fall?"
If the weight stays up, they snip it off.
They keep doing this, step by step, until they reach a point where removing any more paperclips would make the weight drop.
What's left is the Minimal Core: the absolute smallest chain needed to hold the weight.

The Bottom Line

The paper concludes that when AI models "think" out loud, they are often over-talking. They generate long, verbose explanations that contain a lot of removable redundancy.

By finding the Minimal Core, we can:

Compress the thinking (remove 46% of the words).
Clarify the logic (make it easier to see what actually matters).
Verify the thinking (distinguish better between right and wrong answers).

The authors are careful to say this isn't about finding the "human" explanation or the "true" cause of the answer. It's purely about finding the smallest set of steps that keeps the model's answer the same. It's a way of stripping away the noise to see the signal.

Technical Summary: Uncovering the Representation Geometry of Minimal Cores in Overcomplete Reasoning Traces

Problem Statement

Large Language Models (LLMs) increasingly generate long chain-of-thought (CoT) traces to solve complex reasoning problems. However, it remains unclear how much of this generated reasoning is strictly necessary to preserve the model's final prediction. Existing work focuses on eliciting better reasoning or training models to reason more efficiently, but few studies analyze the behavioral sufficiency of already-generated traces.

This paper posits that generated reasoning traces are often overcomplete: they contain a proper subset of steps that preserves the model's predictive behavior, surrounded by "removable reasoning mass" such as restatements, paraphrases, stylistic justifications, redundant checks, or alternative routes that do not affect the final output. The central question is: How much of a generated reasoning trace is actually needed?

Methodology

Definitions and Formalism

The authors formalize the concept of a Minimal Core as the smallest subset of steps $S \subseteq [T]$ from a trace $z = (r_1, \dots, r_T)$ that preserves the model's predictive behavior.

Answer-Preserving Sufficiency: A subset is sufficient if it yields the same top-1 answer as the full trace.
Distribution-Preserving Sufficiency: A stricter condition where the subset preserves the predictive distribution within a tolerance $\epsilon$ (measured via divergence $D$ ).
Metrics:
- Compression Ratio (CR): $|S^*|/T$ , the fraction of steps retained.
- Redundancy Mass (RM): $1 - CR$, the fraction of removable steps.
- Step Necessity ( $\Delta_t$ ): Measured via leave-one-out loss change. A step has high necessity if its removal significantly increases the task loss (e.g., negative log-likelihood of the answer).
- Necessity Concentration: The fraction of total necessity mass accounted for by the top- $k$ steps.

Extraction Algorithms

To identify minimal cores, the paper proposes:

Greedy Backward Elimination: Starting from the full trace, the algorithm iteratively removes the step that causes the least "harm" (measured by a verifier score, confidence, or loss increase) while maintaining the sufficiency predicate. This terminates when no single-step deletion preserves sufficiency, yielding a locally irreducible sufficient subset.
Necessity-Guided Pruning: A cheaper, non-adaptive alternative that ranks steps by necessity and removes low-necessity steps first.
Random Deletion Baseline: Used to establish that the structure of removable steps is not merely a function of trace length.

Theoretical Guarantees

The paper establishes several theoretical properties:

Existence: Minimal cores exist for any finite sufficient trace.
Local Irreducibility: Greedy backward elimination guarantees a locally irreducible sufficient subset (no single step can be removed without violating sufficiency).
Certificates: Observed sufficient deletions serve as certificates of overcompleteness, providing upper bounds on compression ratios and lower bounds on removable mass.

Representation Geometry Analysis

The authors analyze the embedding geometry of full traces, minimal cores, and removed steps using:

Compactness: Variance of embeddings.
Correctness Separation: Ability of linear probes and nearest-neighbor classifiers to distinguish correct vs. incorrect traces.
Intrinsic Dimensionality: Estimated via PCA participation ratios.
Transferability: Performance of cores extracted from one model family when used as context for another.

Experimental Setup

The framework was evaluated across six deliberative reasoning benchmarks:

Arithmetic: GSM8K
Competition Mathematics: MATH500, AIME24, AMC23
Expert Scientific Reasoning: GPQA-Diamond
Commonsense Multi-hop QA: StrategyQA

Models tested included open-weight models (DeepSeek-R1-Distill-Qwen-7B/32B, Qwen3-32B) and the closed-source GPT-5. Traces were generated with low-temperature CoT prompting.

Key Results

1. Substantial Overcompleteness

Across all datasets and models, greedy minimal-core extraction removed a significant portion of the trace while preserving the original answer:

Average Removal: Approximately 46% of steps are removable.
Retention: The original answer is preserved in 86% of cases on average.
Variation: Overcompleteness is highest in GSM8K and StrategyQA (where traces often contain explanatory slack) and lower in AIME24 and GPQA-Diamond (where derivations are more brittle).

2. Concentrated Predictive Support

Predictive necessity is highly sparse:

The top 3 steps account for an average of 65% of the measured necessity mass.
The top 5 steps account for 82%.
This suggests that reasoning utility is concentrated in a small set of high-necessity steps rather than distributed evenly.

3. Transferability and Geometry

Minimal cores exhibit a "cleaner" representation geometry compared to full traces:

Cross-Model Transfer: Minimal cores transfer across model families with 85% off-diagonal answer retention, significantly outperforming random subsets of the same length.
Correctness Separation: Minimal core embeddings improve the separation between correct and incorrect traces by 11 points (probe accuracy) compared to full traces.
Dimensionality: Minimal cores reduce estimated intrinsic dimensionality by 34%, indicating they occupy a more compact region of the representation space.
Removed Steps: Embeddings of removed steps are more diffuse and less predictive of correctness, confirming they carry weaker task-relevant signals.

Significance and Claims

The paper claims that full reasoning traces are often verbose and overcomplete, containing removable scaffolding that does not contribute to the model's predictive behavior. By isolating minimal cores, the authors demonstrate that:

Behavioral Sufficiency: The effective support for language model predictions is sparse and concentrated in a small subset of steps.
Geometric Purity: Minimal cores expose a cleaner, lower-dimensional, and more correctness-informative geometry of reasoning than full traces.
Transferable Structure: The core reasoning steps are robust across different model families, suggesting they capture generalizable reasoning patterns rather than model-specific phrasing.

The authors explicitly note limitations: minimal cores are behavioral and model-relative, not complete causal explanations or human rationales. The greedy extraction is locally, not globally, optimal. Furthermore, likelihood-based metrics require open-weight models, limiting some analyses to behavioral evaluations for closed-source models.

In conclusion, the work provides a quantitative framework for understanding the "effective support" underlying LLM predictions, suggesting that while long traces aid readability and verification, the core predictive machinery relies on a significantly smaller, geometrically distinct subset of reasoning steps.