💬 NLP

Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMs

This paper introduces an axiomatic framework comprising four functional metrics to evaluate latent thought representations in LLMs independently of downstream accuracy, revealing that current models structurally fail to satisfy these axioms by encoding minimal information beyond input embeddings and lacking the ability to distinguish between specific questions within the same task.

Original authors: Fahd Seddik, Fatemeh Fard

Published 2026-06-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Fahd Seddik, Fatemeh Fard

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) as a brilliant but very private chef. When you ask it to solve a math problem or answer a tricky question, it doesn't just spit out the answer immediately. Instead, it goes through a "thinking" process.

In the past, we could see this thinking because the chef wrote it down step-by-step (like a "Chain of Thought"). But recently, developers have been trying to make the chef think faster by hiding those steps inside a "black box" of continuous, invisible numbers (called latent thought representations). The idea is: "If the final answer is correct, the hidden thinking must be good, right?"

The Problem:
The authors of this paper say, "Not so fast." Just because the chef serves a delicious meal (a correct answer) doesn't mean the hidden thinking process was actually logical. The chef might be guessing, or the "thinking" might be a messy jumble that only works by accident.

To fix this, the authors created a new way to inspect the chef's hidden thoughts without looking at the final answer. They built a "Four-Point Inspection Checklist" (called Axioms) to see if the hidden thoughts are actually doing their job.

The Four-Point Inspection Checklist

Think of the "Thought" as a secret note the chef writes before cooking. The paper argues a good secret note must pass four tests:

Causality (The "Cause and Effect" Test):
- The Metaphor: If you replace the chef's written recipe with their secret note, should the final dish still taste the same?
- The Claim: The secret note must contain exactly the same information as the reasoning steps. If you swap the steps for the note, the model should still be able to predict the answer perfectly. If the note is missing a crucial ingredient, the dish fails.
Minimality (The "No Fluff" Test):
- The Metaphor: Imagine the chef is given a 50-page story about a cat, but the question is only about the weather. A good secret note should only contain the weather info, not the whole story about the cat.
- The Claim: The thought should be a compressed, efficient summary. It shouldn't carry extra "noise" or irrelevant details from the prompt that don't help solve the problem.
Separability (The "Distinct Identity" Test):
- The Metaphor: If you give the chef two very similar math problems (e.g., "What is 2+2?" and "What is 2+3?"), their secret notes should look totally different. If the notes for both problems look identical, the chef isn't actually distinguishing between the questions; they are just guessing.
- The Claim: The hidden thought must be unique for every specific question. It needs to clearly separate one problem from another, even if they look similar.
Stability (The "Consistency" Test):
- The Metaphor: If the chef says "The answer is four" in one sentence and "It's four" in another, the secret note should be the same. Also, if the chef is confused and sometimes says "Yes" and sometimes "No," the secret note should reflect that confusion, not just pick one side randomly.
- The Claim: The thought shouldn't change just because the wording changes slightly. It should also accurately reflect how uncertain the model is.

What They Found (The Audit)

The researchers took five different popular AI models (like Llama and DeepSeek) and put them through 23 different reasoning tasks (like spatial puzzles, logic games, and math). They checked the "secret notes" these models generated against their four-point checklist.

The Shocking Result:
None of the models passed all four tests.

They could tell the difference between tasks: The models could tell the difference between a "Math" question and a "History" question.
But they failed the fine details: When asked two different questions within the same task (e.g., two different math problems), the models' "secret notes" looked almost identical. They collapsed into a generic blob.
The Prompt was better than the Thought: Surprisingly, the simple text of the original question (the prompt) was often a better "thought representation" than the complex hidden numbers the model generated. The hidden thoughts didn't add much new information; they were mostly just echoing the input.

The Big Conclusion

The paper concludes that this failure isn't because the models are too small or trained poorly. It's a structural problem. Even the biggest, most advanced models are currently failing to create a true, distinct "thought" for every single question. They are good at producing the right answer, but their internal "thinking" process is often a collapsed, generic blur that doesn't actually distinguish between specific problems.

In short: The models are getting the right answers, but their "thinking" is broken in a way that standard tests can't see. The authors' new checklist is the tool needed to find these hidden flaws.

Technical Summary: Formalizing Latent Thoughts: Four Axioms of Thought Representation in LLMs

Problem Statement

The field of Large Language Model (LLM) reasoning is increasingly shifting from discrete Chain-of-Thought (CoT) tokens toward continuous latent representations (e.g., Soft Thinking, Latent Thinking) to improve efficiency and reasoning capacity. However, the evaluation of these "continuous thought representations" remains fundamentally flawed. Current methodologies almost exclusively rely on downstream task accuracy (e.g., benchmark pass rates) to judge representational quality. This approach conflates the quality of the internal representation with the model's overall capacity to decode or process that representation. Consequently, representational failures are masked by downstream accuracy; a model may achieve high benchmark scores while its internal "thoughts" fail to faithfully encode the necessary reasoning steps or distinguish between distinct instances of the same task.

There is a lack of:

A principled definition of what functional properties a valid thought representation must possess.
An intrinsic evaluation protocol that measures representation quality independently of downstream task accuracy.
A clear understanding of the current status of candidate methods, given reports that latent reasoning tokens are often unnecessary for predictions or fail to implement structured latent-space searches.

Methodology

The authors propose an axiomatic evaluation framework that characterizes thought representations based on four functional properties, defined independently of any downstream decoding process. The framework treats the thought representation ( $T$ ) as a latent functional state mediating the transformation from an input space ( $X$ ) to a semantic output space ( $S$ ).

The Four Axioms

The framework defines four axioms, each with a corresponding quantitative measure computed directly on the source LLM without retraining:

Causality: $T$ must functionally substitute the explicit reasoning prefix ( $y_{pre}$ ) within the model's computational graph.
- Metric: KL Substitution Error. Measured by replacing the token embeddings of $y_{pre}$ with the projected $T$ and calculating the Kullback-Leibler (KL) divergence between the original continuation distribution $P(y_{suf} | y_{pre})$ and the intervened distribution $P(y_{suf} | T)$ . Lower values indicate better causality.
Minimality: $T$ must compress the input while retaining maximum relevance to the output distribution, adhering to the Information Bottleneck principle. It should filter out nuisance variables (irrelevant context) present in the input.
- Metric: Minimality Gap ( $\Delta_{IB}$ ). A cross-entropy surrogate for the Information Bottleneck Lagrangian at $\beta=2$ . It is calculated as $\Delta_{IB} = CE(X | Y, T) - CE(Y | T)$ . Higher values indicate a representation that is sufficient for the output but adds minimal residual information about the input beyond what the output already provides.
Separability: $T$ must possess sufficient topological structure to distinguish between semantically non-equivalent output distributions. It must be able to differentiate between two distinct questions within the same task (fine-grained) and between different tasks (coarse-grained).
- Metric: Discriminator Accuracy. A trainable binary discriminator is trained to align $T$ with a candidate output sequence. Accuracy is measured on same-task pairs (fine-grained) and cross-task pairs (coarse-grained).
Stability: $T$ must be invariant to surface-level lexical variations (paraphrasing) and robust to sampling stochasticity (mode collapse). It should encode the parameters of the semantic distribution $P(S|x)$ rather than a single realization.
- Metric: Distributional Consistency Score (DCS). The AUROC of a probe predicting whether a question's beam outputs span multiple semantic equivalence classes (semantic entropy $H_x > 0$ ).

Experimental Setup

Candidates: The study audits several candidate representations, including Last Input Token (LIT) hidden states, Soft Thinking (with and without Gumbel noise), and Latent Thinking, across varying "thinking budgets" (1 to 128 steps).
Models: Five open-weight LLMs spanning diverse architectures and training paradigms: Llama-3.1-8B, Llama-3.3-70B (Dense/Instruct), DeepSeek-R1-Distill-Qwen-32B (Reasoning-distill), Skywork-OR1-32B (Native RL), and GPT-OSS-20B (Sparse MoE).
Tasks: 23 reasoning tasks from the Big Bench Extra Hard (BBEH) dataset, covering spatial reasoning, factual QA, logical puzzles, and more.
Protocol: Evaluations are performed on the source LLM using a frozen 1B backbone for probing, ensuring the metrics are decoupled from the source model's size and downstream performance.

Key Results

The application of this intrinsic protocol reveals a "representational collapse" that downstream benchmark accuracy fails to detect:

No Candidate Satisfies All Axioms: No evaluated candidate representation simultaneously satisfies Causality, Minimality, Separability, and Stability.
Separability Failure (Per-Question Identity): While representations reliably distinguish between different tasks (cross-task accuracy is near saturation), they fail to distinguish between different instances of the same task (same-task accuracy). Almost all candidates (except Output Embeddings) perform at or near the random baseline for same-task discrimination. This indicates that latent representations lose the specific identity of individual questions, collapsing distinct reasoning paths into indistinguishable vectors.
Input Embedding Competitiveness: The input prompt embedding itself is competitive with, and often superior to, candidate thought representations across all four axes. This suggests that current latent representations encode little information beyond what is already present in the input embedding.
Structural Failure: The failure pattern is consistent across dense, reasoning-distilled, and RL-trained model families. Increasing the number of thinking steps (e.g., in Soft Thinking or Latent Thinking) does not resolve the collapse; in some cases, performance degrades as step count increases.
Causality vs. Input: No thought representation consistently provides more causal information than the input embedding alone. Replacing the reasoning prefix with $T$ yields a continuation distribution indistinguishable from replacing it with the input embedding, implying $T$ adds no unique causal signal.

Significance and Claims

The paper claims to provide a principled means to develop and audit candidate thought representations as objects of study, evaluated for what an LLM encodes about a problem rather than for the downstream reasoning quality of any subsequent chain.

Diagnostic Resolution: The framework allows researchers to attribute downstream accuracy changes to specific representational properties (e.g., a drop in Separability) rather than aggregate benchmark scores that obscure the root cause.
Optimization Targets: The four measures provide explicit, quantifiable optimization targets for future research, moving beyond heuristic proxies like token budgets or imitation of explicit CoT.
Structural Insight: The findings suggest that the gap between current latent reasoning methods and ideal functional thought is structural, not merely a function of model size or specific training procedures (like RL or distillation). The current generation of latent representations fails to implement the structured, instance-specific search required for robust reasoning.

The authors position this framework as orthogonal to reasoning benchmark evaluation, offering a necessary layer of analysis to understand the internal mechanics of reasoning models before they can be reliably improved. The code and evaluation pipeline are made publicly available to facilitate further audit and development.