💬 NLP

States Hidden in Hidden States: Implicit Discrete State Representations Emerge in LLMs' Hidden States

This paper investigates how Large Language Models perform extended calculations without explicit chain-of-thought reasoning by forming and utilizing Imperfect Implicit Discrete State Representations within their hidden states, while also characterizing their formation and identifying their contribution to computational errors.

Original authors: Junhao Chen, Shengding Hu, Zhiyuan Liu, Maosong Sun

Published 2026-08-05

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Junhao Chen, Shengding Hu, Zhiyuan Liu, Maosong Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a magician pull a rabbit out of a hat. You know the rabbit didn't just appear from thin air, but you can't see the trick. For a long time, scientists studying Artificial Intelligence (AI) have been in a similar boat. They know these giant computer brains, called Large Language Models (LLMs), can do amazing things like write poems, solve math problems, and chat like humans. But how they do it inside their "mind" (which is just a massive collection of numbers and layers) has been a bit of a black box.

To understand this paper, you need to know two simple things. First, LLMs work by predicting the next word in a sentence, one word at a time. Second, when they do this, they create a temporary "scratchpad" of information called a hidden state. Think of this hidden state like a mental note the AI makes while it's thinking. Usually, we assume the AI just reads the whole question and then guesses the answer. But what if, while reading a long list of numbers, the AI is secretly keeping a running total in its head, just like you do when you add up your grocery bill in your head without writing anything down? This paper asks: Is the AI actually doing that mental math, or is it just guessing?

The Magic Trick of Mental Math

Imagine you are at a party, and someone hands you a long list of numbers: 17, 38, 32, 87... and asks you to add them all up instantly, without writing anything down or saying the steps out loud. Most people would struggle, but the smartest AI models in the world are starting to get really good at this. They can look at a string of up to 15 numbers and spit out the correct total immediately, without showing their work.

This paper, titled "States Hidden in Hidden States," investigates exactly how these AI models pull off this trick. The researchers from Tsinghua University hypothesized that these models aren't just guessing; they are secretly building Implicit Discrete State Representations (IDSRs).

Let's break that fancy term down. Imagine the AI is a team of workers passing a bucket of water down a line.

Implicit: They aren't shouting the number out loud (generating text).
Discrete State: They are holding a specific, exact number in their hands (like "37" after adding 17 and 20).
Representation: This number exists as a pattern inside the AI's internal layers.

The researchers wanted to prove that the AI is actually holding onto these intermediate numbers as it reads the list, rather than waiting until the very end to try and calculate everything at once.

The Detective Work: Peeking Inside the Brain

To see if this "mental scratchpad" really exists, the researchers played a game of "20 Questions" with the AI's internal layers. They used a tool called a probe. Think of a probe like a tiny, super-fast detective that can peek at the AI's internal notes at any specific moment and ask, "What number are you thinking of right now?"

They tested this on a variety of models, from smaller ones (like Llama2-7B) to the giants (like GPT-4 and Qwen-72B). They asked the models to solve addition problems with anywhere from 2 to 14 numbers.

What they found:

The Big Get Bigger: The ability to do this mental math "emerges" as models get bigger. Small models can add two or three numbers, but once the list hits eight numbers, they crash. The biggest models, however, can handle lists of 11 or even 15 numbers with surprising accuracy.
The Hidden Note is Real: When the researchers used their "detective probes" on the AI's hidden layers, they found that the models were indeed holding the running totals. For example, after the AI read "17 + 24," the internal state of the model contained a representation of "41," even though it hadn't said the word "41" out loud yet.
It's Not Perfect: The bad news is that this mental note isn't perfect. As the list of numbers gets longer, the "note" gets a bit fuzzy. The researchers found that while the AI can hold the number perfectly for the first few steps, by the time it gets to the 15th number, the internal representation starts to lose some detail. This "lossy" compression is why even the smartest models sometimes get the final answer wrong.

How the AI Builds the Number

The paper also figured out how the AI builds these numbers, which is surprisingly similar to how humans do it.

One Digit at a Time: The AI doesn't calculate the whole number all at once. It builds the answer digit by digit, starting from the right (the ones place), then the tens, then the hundreds. It's like adding columns on paper, but happening entirely inside the AI's brain.
The "Shallow" vs. "Deep" Layers: The AI has many layers of processing. The researchers found that the first 10 layers are like the "calculator" part of the brain—they just do the raw math. But as the information moves to the deeper layers (around layer 10 and beyond), the AI starts to mix in the context of the sentence. It's like the calculator getting distracted by the conversation around it. This mixing is what causes the "fuzziness" or errors in the final answer.
Linear, Not Magical: The AI doesn't do the math in a magical, all-at-once way. It processes the list in a straight line. If you have numbers A, B, C, and D, it calculates (A+B), then takes that result and adds C, then adds D. It doesn't do (A+B) and (C+D) separately and then combine them.

The "Bridge" Experiment

To prove the AI was actually using these hidden notes and not just guessing, the researchers did a clever trick. They built a "bridge" in the AI's attention mechanism. Imagine the AI is reading a long sentence, and they blocked it from looking back at the beginning of the sentence, forcing it to only look at the very last thing it read.

If the AI was just guessing based on the whole sentence, this block would break it completely. But, the AI could still solve the math problem if it could "see" the bridge (the most recent addition sign). This proved that the AI was carrying the running total forward from one step to the next, just like a human would.

The Takeaway

This paper suggests that when giant AI models do math without showing their work, they are indeed doing the work. They are creating invisible, internal "sticky notes" that hold the running total as they read.

However, there is a catch. These sticky notes aren't perfect. As the list of numbers gets longer, the notes get a little smudged, and the AI starts to make mistakes. The researchers believe that if we can figure out how to make these internal notes clearer and less "lossy," we might be able to make AI much better at complex reasoning tasks, not just math.

For now, we know the AI isn't magic; it's just really good at keeping a mental tally, even if that tally sometimes gets a little blurry by the end of a long list.

Technical Summary: States Hidden in Hidden States

Problem Statement

Large Language Models (LLMs) have demonstrated emergent abilities to perform arithmetic calculations, specifically iterative additions, without generating intermediate chain-of-thought (CoT) tokens. While prior research has focused on binary arithmetic or the general interpretability of mathematical tasks, the mechanism by which models track intermediate results in multi-step iterative reasoning remains unclear. Specifically, it is unknown whether models memorize training data, rely on explicit reasoning, or form internal representations of intermediate states to solve these problems directly. This paper investigates the hypothesis that LLMs form Implicit Discrete State Representations (IDSRs) within their hidden states to facilitate sequential symbolic calculation.

Methodology

The authors employ a combination of synthetic dataset generation and probing techniques to validate the existence and properties of IDSRs.

1. Dataset Construction

A synthetic dataset of 131,300 iterative addition and subtraction problems was created. The dataset includes:

Variable Length: Sequences ranging from 2 to 14 operands.
Operands: Integers sampled uniformly from 1 to 100 (1 to 3 digits).
Prompt Diversity: Three prompt types were used: standard addition, subtraction, and "prompting" (where the prompt instructs the model to ignore the formula or answer with a specific token like "apple").
Split: The data was partitioned into training (80%), validation (10%), and test (10%) sets for probe training.

2. Experimental Setup

Models Evaluated: A diverse range of open-source (e.g., Llama2, MiniCPM, Mistral, Qwen, DeepSeek) and closed-source (e.g., GPT-3.5, GPT-4, Claude) models were tested.
Hidden State Extraction: During inference, hidden states ( $H_{i,j}$ ) corresponding specifically to operator tokens (e.g., the "+" signs) were extracted to avoid bias from operand representations.
Probing Mechanism: Multi-layer perceptrons (MLPs) and single-layer perceptrons were trained to classify the hidden states. The probes predicted:
- Whole Number Probing: The total sum as a whole number.
- Digit-wise Probing: Individual digits (units, tens, hundreds) of the intermediate sum.
Intervention Experiments: An "Attention Bridge" experiment was conducted where attention masks prevented subsequent tokens from attending to earlier operands, forcing them to rely solely on the information passed through a specific "bridge" token (the first addition sign).

Key Findings and Results

1. Emergence of Implicit Computation

Scale Correlation: There is a strong correlation between model size and the ability to perform iterative addition. Smaller models (e.g., 7B) show rapid accuracy degradation as sequence length increases (near zero accuracy at 8 operands), while larger models (e.g., 72B, GPT-4) maintain >50% accuracy for sequences up to 8 operands and non-zero performance for 11+ operands.
Non-Memorization: The capability is unlikely to be memorized, as the parameter space required to store all possible sums of long sequences ( $O(99^L)$ ) exceeds model capacity, and the task offers negligible utility for general pretraining optimization.

2. Existence and Properties of IDSRs

Existence: Probing experiments confirmed that hidden states at operator tokens contain information about the current intermediate sum. Prediction accuracies significantly exceeded random chance across all models.
Lossy Transmission: The transmission of IDSRs is not lossless. Accuracy drops as the sequence progresses (e.g., from 100% at the second "+" to 37% at the final "=" in Qwen-72B), contributing to final output errors.
Linearity vs. Non-linearity:
- Shallow Layers (0–10): Representations are largely linear and task-agnostic ("shallow-semantic"). They compute arithmetic values regardless of the specific prompt context (e.g., subtraction vs. addition).
- Deep Layers (>10): Representations transition to non-linear features ("semantic layers"). These layers integrate task context, which reinitializes the IDSR formation but reduces resolution compared to shallow layers.
Digit-Level Formation: IDSRs form independently for each digit. The order in which digits become predictable follows the human calculation order (units $\to$ tens $\to$ hundreds), suggesting models perform multi-digit addition via iterative single-digit additions.

3. Utilization of IDSRs

Sequential Processing: Models utilize IDSRs sequentially rather than in a logarithmic (tree-like) fashion. Probing experiments showed that the accuracy for a sum of $n$ numbers peaks only after the accuracy for the sum of $n-1$ numbers has peaked, ruling out parallel combination strategies.
Attention Bridge Validation: When direct attention to early operands was masked, models retained non-trivial accuracy if information could pass through the "bridge" token. This confirms that models rely on the intermediate state encoded at operator tokens to compute subsequent steps.

Key Contributions

Hypothesis Validation: The paper provides empirical evidence that LLMs form Implicit Discrete State Representations (IDSRs) in hidden states to perform iterative arithmetic without explicit CoT.
Characterization of Formation: The study maps the formation of IDSRs across three perspectives:
- Digit-wise: Independent, sequential formation starting from the least significant digit.
- Layer-wise: A transition from linear, task-agnostic arithmetic computation in shallow layers to non-linear, context-aware semantic processing in deeper layers.
- Sequence-wise: Information propagation along the sequence, where later tokens utilize the state encoded in earlier operator tokens.
Mechanism of Failure: The authors identify that the "lossy" nature of IDSR transmission in current open-source models is a primary contributor to calculation errors in long sequences.

Significance and Claims

The authors position this work as an initial exploration into the internal mechanisms of LLMs regarding symbolic calculation and multi-step reasoning. The significance lies in:

Interpretability: Offering a concrete example of how LLMs perform "intrinsic iterative reasoning" by tracking discrete states, serving as a foundational step for understanding complex reasoning capabilities.
Model Limitations: Highlighting that current models are not "lossless" in their state tracking, which explains why they fail on longer sequences despite emergent abilities.
Future Directions: The paper suggests that understanding and potentially reducing the loss in IDSR generation could enhance the mathematical and reasoning capabilities of LLMs. It explicitly notes that the findings are currently validated on synthetic mathematical datasets and that the capability is an emergent property dependent on model scale.

The paper concludes by acknowledging limitations, such as the lack of diversity in task types beyond synthetic math, and calls for future research into the factors influencing IDSR resolution and the scalability of these representations across different model sizes.