💬 NLP

Reasoning Models Know What's Important, and Encode It in Their Activations

This paper demonstrates that language models encode an internal, generalizable representation of reasoning step importance within their activations, revealing that analyzing model internals provides deeper insights into reasoning processes than surface-level token analysis.

Original authors: Yaniv Nikankin, Martin Tutek, Tomer Ashuach, Jonathan Rosenfeld, Yonatan Belinkov

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yaniv Nikankin, Martin Tutek, Tomer Ashuach, Jonathan Rosenfeld, Yonatan Belinkov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: The "Hidden Brain" vs. The "Spoken Words"

Imagine you are watching a magician perform a complex trick. They talk through every single step: "I'm picking up the card, I'm shuffling the deck, I'm thinking about the color, I'm checking the time..."

To an observer, it looks like every single word they say is part of the magic. But in reality, the magician might only need three specific moves to make the trick work. The rest? Just filler, nervous chatter, or redundant explanations.

This paper asks a simple question about AI (Large Language Models): When an AI "thinks" through a math problem, does it actually know which steps are the magic moves and which are just chatter?

The researchers found something fascinating: The AI knows exactly what's important, but it doesn't always say it out loud.

Analogy 1: The "Ghost in the Machine" (Activations vs. Tokens)

When an AI solves a problem, it does two things:

Tokens (The Spoken Words): The actual text it generates (e.g., "Let's calculate 5 times 5...").
Activations (The Ghost): The internal electrical signals and math happening inside the AI's "brain" before it types a single letter.

The Paper's Discovery:
If you try to figure out which steps are important just by reading the AI's words (tokens), you get confused. The AI might say a lot of fluff that looks important but isn't.

However, if you look at the internal signals (activations), the AI is screaming, "This step is CRITICAL!" or "This step is useless, ignore it!"

The Metaphor:
Think of the AI's words as a student giving a presentation. They might ramble, repeat themselves, and say "um" a lot.
Think of the AI's activations as the student's brain. Even while they are rambling, their brain is quietly organizing the notes, knowing exactly which facts are the "key points" and which are just "filler."

The researchers built a "lie detector" (called a probe) that listens to the AI's brain signals. They found that the AI can predict, before it even writes the next sentence, whether the current sentence is vital for the answer or if it could be deleted without breaking the logic.

Analogy 2: The "Core vs. The Fluff"

The researchers tested this by trying to delete steps from the AI's reasoning chain.

The Token Approach (Reading the script): They asked a smart human (or another AI) to read the text and decide what to cut. This was often wrong. They kept too much fluff and sometimes cut important stuff.
The Activation Approach (Listening to the brain): They used the internal signals to decide what to cut. This was incredibly precise. They found that they could delete 30% to 50% of the steps, and the AI would still get the right answer.

The Metaphor:
Imagine a long, winding road trip.

The Tokens are the GPS voice saying, "Turn left, drive 5 miles, pass a gas station, drive 2 miles, turn right..."
The Activations are the map in the driver's head.
The researchers found that the driver (the AI) knows that the "gas station" and "drive 2 miles" parts are just filler. If you remove them from the map, the driver still knows exactly how to get to the destination. The AI's internal map is much more efficient than the spoken directions.

Analogy 3: The "Universal Language of Importance"

One of the coolest findings is that this "sense of importance" isn't unique to one specific AI.

The Metaphor:
Imagine two different people, Alice and Bob, solving the same math problem. They might use different words, different sentence structures, and different styles.

Alice says: "First, I multiply the numbers."
Bob says: "Okay, let's crunch the digits."

If you train a detector on Alice's brain signals to find the "important steps," and then you use that same detector on Bob's brain signals, it still works!

This suggests that "importance" is a fundamental concept that all these reasoning models share. It's like a universal language of logic that exists deep inside their code, regardless of how they talk on the outside.

Why Does This Matter?

Efficiency: If we know which steps are "fluff," we could teach AI to skip them. This would make them faster and cheaper to run.
Trust (Faithfulness): Sometimes AI gives a wrong answer but writes a long, convincing explanation that looks right. This paper shows that the AI's brain might actually know the explanation is nonsense (removable) even if it writes it anyway. This helps us spot when AI is "hallucinating" or lying.
Better Understanding: It tells us that to truly understand how AI thinks, we can't just read its output. We have to look under the hood at its internal signals.

The Bottom Line

The paper concludes that AI models are smarter than their words suggest. They have a hidden, internal understanding of what matters in a reasoning chain. They know which steps are the "skeleton" of the argument and which are just "flesh" that can be trimmed away.

To truly understand AI reasoning, we need to stop just reading the transcript and start listening to the internal signals. The AI knows what's important; it's just waiting for us to learn how to listen.

1. Problem Statement

Large Language Models (LLMs) solve complex tasks by generating long chains of reasoning (Chain-of-Thought). However, not all steps in these chains are equally critical; some are essential for deriving the correct answer, while others are redundant, performative, or "filler."

The Core Question: How can we accurately identify which reasoning steps are truly important?
The Gap: Existing approaches rely on analyzing the tokens (the generated text) or using external LLM judges to determine importance. The authors hypothesize that these surface-level approaches may miss crucial information, whereas the model's internal activations might contain a more faithful and precise representation of step importance.
Goal: To determine if models encode an internal notion of "step importance" in their latent activations, and whether this signal is superior to token-based analysis for identifying the minimal sufficient reasoning path.

2. Methodology

A. Defining Importance via Removability

The authors operationalize "importance" through causal removability. A step is deemed "important" if removing it prevents the model from generating the correct answer.

Sufficiency: The ability of a reasoning subsequence to produce the ground-truth answer (measured by sampling completions).
Attainability: The probability of the subsequence occurring naturally given the prompt (measured via perplexity), ensuring the remaining chain isn't an artificial, degenerate sequence.
Core Reasoning Subsequence ( $T^*$ ): The shortest subsequence that satisfies both sufficiency and attainability thresholds.

B. Identifying Important Steps (The Greedy Procedure)

To find $T^*$ , the authors compare three methods for ranking and removing steps:

Activation-based (Proposed): Uses gradient attribution to calculate the influence of each step on subsequent steps and the final answer. Steps with low cumulative influence are removed iteratively.
Token-based (Baseline): Uses an external LLM (Gemini-2.5-Pro) to judge which steps influence the answer and rank them by importance.
Random (Baseline): Randomly ranks steps for removal.

C. Probing for Internal Representations

Once steps are labeled as "important" or "removable" (using the activation-based method as the ground truth), the authors train classification probes to see if the model already knows this distinction:

Input: Latent activations of a specific reasoning step (averaged over tokens and layers).
Task: Binary classification (Removable vs. Important).
Conditions:
- Full Context: The probe sees the step's activation plus all preceding context (input problem + previous reasoning steps).
- Context-Free: The probe sees only the step in isolation.
- Baselines: Comparison against TF-IDF, semantic embeddings (Sentence-BERT), and the external LLM judge.

D. Cross-Model and Surface Analysis

Cross-Model Transfer: Probes trained on Model A's activations are tested on Model B's activations to see if the concept of importance is universal.
Surface Feature Correlation: The authors test if importance correlates with surface features like step length, position, or specific keywords (e.g., "let's think").

3. Key Contributions

Superiority of Latent Activations: The paper demonstrates that activation-based methods consistently identify a smaller, more precise set of core reasoning steps compared to token-based methods (LLM judges) or random baselines. This suggests latent information captures reasoning structure better than surface text.
Pre-hoc Encoding of Importance: Crucially, the authors show that models encode the importance of a step before generating subsequent steps. A probe trained on the current step's activations (given the history) can predict its future importance with high accuracy (>80%), indicating the model "knows" what matters internally before it speaks.
Universality and Distribution:
- Cross-Model Generalization: Importance signals are not model-specific. A probe trained on one model's activations can accurately predict importance in a different model's reasoning traces.
- Distributed Representation: Importance is not localized to a single layer or token position; it is distributed across the residual stream and requires aggregating information from all layers to be detected accurately.
Decoupling from Surface Features: The internal representation of importance has low correlation with surface-level features (length, position, specific keywords). This proves that the model's internal "knowledge" of importance is distinct from the text it generates.

4. Key Results

Core Subsequence Size: The activation-based method reduced the core reasoning subsequence to ~22–30% of the total prefix length across various models (GPT-OSS, DeepSeek-R1, OLMo), whereas token-based methods resulted in larger, less precise subsets (~30–40%).
Probe Accuracy:
- Full-Context Activations: Achieved >80% accuracy in predicting step importance.
- Token-Based/LLM Judge: Achieved only ~60% accuracy, even with full context.
- Context-Free: Accuracy dropped significantly (10–20% drop), proving that context is essential for the model to evaluate importance.
Cross-Model Agreement: When transferring probes between models, agreement ratios remained high (often >80%), confirming that the concept of "important step" is a universal property of reasoning models, not an artifact of a specific architecture.
Surface Feature Failure: Regression models trained on surface features (length, position, filler words) performed significantly worse than activation probes, confirming that importance is a latent, non-surface concept.

5. Significance and Implications

Faithfulness of Reasoning: The findings suggest that generated reasoning chains are often unfaithful to the model's internal computation. Steps that appear critical in the text may be internally marked as removable, and vice versa.
New Paradigm for Interpretability: To truly understand how LLMs reason, researchers must look beyond tokens and into model internals (activations). Surface-level analysis fundamentally misses the "why" behind the reasoning.
Efficiency and Pruning: Since models encode importance internally, it may be possible to use these signals at inference time to prune reasoning chains dynamically, removing redundant steps to reduce latency and context costs without sacrificing accuracy.
Universal Reasoning Mechanisms: The cross-model consistency implies that the ability to distinguish important from unimportant steps is a fundamental capability of reasoning models, regardless of their specific training data or architecture.

In conclusion, the paper establishes that reasoning models possess a robust, internal, and universal representation of step importance that is invisible to surface-level analysis but clearly encoded in their latent activations.