🤖 machine learning

Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs

This paper demonstrates that large language models' failure to count repeated tokens stems not from an inability to represent the correct count internally, but from a specific format-triggered MLP mechanism that overwrites the accurate representation with a fixed wrong answer during output generation.

Original authors: Sohan Venkatesh

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sohan Venkatesh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read librarian (the AI model). You ask this librarian to count how many times the word "apple" appears in a long list. You'd expect them to get it right, but instead, they confidently say, "There are 8 apples," even though there are clearly 10.

For a long time, people thought the librarian was just bad at counting. They assumed the librarian's brain didn't have a "counter" built into it, so they couldn't keep track of the numbers.

This paper says that assumption is wrong.

Here is the simple breakdown of what the researchers actually found, using some everyday analogies:

1. The Librarian Does Know the Answer

The researchers looked deep inside the librarian's brain (the model's internal layers) at every single step of the thinking process. They found that the correct number (10) was written down clearly and perfectly from the very first step to the very last.

The Analogy: Imagine the librarian is walking down a hallway with a whiteboard in their hand. At every single step of the walk, the whiteboard clearly says "10." The information is there, loud and clear. The librarian isn't confused; they actually know the answer.

2. The "Glitchy" Sign-Writer

So, if the librarian knows the answer is 10, why do they say "8"?

The researchers found a specific "glitch" in the middle of the hallway. About 90% of the way through the thinking process, there is a specific worker (a part of the computer code called an MLP block) who sees the list format (words separated by spaces) and decides to ignore the whiteboard.

The Analogy: Imagine the librarian is walking down the hall, holding the "10" sign. Suddenly, a specific worker (let's call him "Format-Fred") jumps out. Fred sees that the list is written with spaces (like "apple apple apple"). He has a pre-programmed rule: "If I see a list of words with spaces, I must change the sign to say '8'."
Fred grabs the whiteboard, erases the "10," and writes "8" on it.
Even though the librarian's brain still knows it's 10 (the original signal is still there in the background), the final output is now "8" because Fred overwrote it.

3. It's Not About the Words, It's About the Format

This glitch only happens with words (like "apple" or "cat"), not numbers (like "1" or "2").

The Analogy: If you give the librarian a list of numbers ("1 1 1 1..."), Format-Fred doesn't care. He only reacts to words. The librarian counts the numbers perfectly. But if you give them words, Fred gets triggered.
Also, if you change the format to use commas instead of spaces (e.g., "apple, apple, apple"), Fred doesn't jump out. The "8" rule isn't triggered, and the librarian gets it right. This proves the problem isn't the counting ability; it's a specific reaction to how the list looks.

4. The "Routing" Problem, Not a "Representation" Problem

The paper makes a big distinction between two types of failures:

Representation Failure: The librarian doesn't know how to count. (This paper says this is NOT happening).
Routing Failure: The librarian knows how to count, but the signal gets hijacked before it reaches the mouth. (This is what is happening).
The Analogy: It's like a radio station playing the correct song (the count), but a DJ in the booth (the MLP block) decides to cut the music and play a different song (the wrong number) right before it goes out to the speakers. The music is fine; the routing is broken.

5. Bigger Models Have a Different Glitch

The researchers also looked at bigger, smarter models. These models usually get the count right, but they fail if you sneak a "banana" into the list of apples.

The Glitch: These bigger models see the banana. Their attention system notices it. But, just like the smaller models, they fail to let that information change the final answer. They see the anomaly but can't "route" that new information to the final output.

The Bottom Line

The paper concludes that these AI models aren't "bad at math." They are actually very good at tracking the numbers internally. The problem is that a specific part of their brain gets triggered by the way the list is written (spaces vs. commas, words vs. numbers) and forces a wrong answer onto the final output, overwriting the correct one that was already there.

To fix this, we don't need to teach the AI how to count (it already knows). We need to fix the "traffic cop" (the routing mechanism) that is currently blocking the correct answer from getting out.

Technical Summary: Repeated-Token Counting Reveals a Dissociation Between Representations and Outputs

Problem Statement
Large language models (LLMs) exhibit systematic and reproducible failures when counting repeated tokens, despite demonstrating strong performance on broader reasoning benchmarks. For instance, when asked to count ten repeated words (e.g., "apple"), models often output a confident, deterministic wrong answer (e.g., "8" or "14") with zero variance under greedy decoding. Common attributions for this failure suggest a lack of internal counting representations or limitations in tracking counts within the model's state. This paper challenges that attribution, proposing instead that the failure lies not in representation but in the routing of information to the output.

Methodology
The authors employ mechanistic interpretability techniques across five instruction-tuned decoder-only models: Llama-3.2 (1B, 3B) and Qwen2.5 (1.5B, 3B, 7B). The experimental setup involves three conditions:

P1 (Baseline): Counting ten identical tokens.
P2 (Anomaly): Counting nine identical tokens with one intruder (e.g., "banana").
P3 (Control): Counting ten unique tokens.

The study utilizes:

Linear Probes: Training Ridge regression probes on the residual stream at every layer to decode the token count, testing whether the correct count is linearly decodable even when the output is wrong.
Logit Lens: Projecting hidden states through the final layernorm and unembedding matrix at each layer to observe when the model "commits" to a specific digit token.
MLP vs. Attention Decomposition: Isolating the contributions of attention sublayers and MLP blocks to determine which component writes or erases specific values in the residual stream.
Causal Interventions: Zero-ablation of specific MLP blocks to test causal involvement.
Format and Token Variation: Testing space-separated vs. comma-separated delimiters and substituting word tokens with digit tokens to analyze format sensitivity and token-type specificity.

Key Results

Representation-Behavior Dissociation:
Linear probes reveal that the correct count is linearly decodable with near-perfect accuracy ( $R^2 > 0.99$ ) from the first transformer layer onward, even in models that output incorrect counts. Crucially, the count is encoded more cleanly in the condition where the model fails (repeated tokens) than in the condition where it succeeds (unique tokens). This confirms the model possesses the correct internal representation but fails to express it.
The Routing Failure (MLP Overwrite):
The failure is caused by a specific, format-triggered Multi-Layer Perceptron (MLP) block located at approximately 88–93% of the network depth (e.g., Layer 14 in Llama-1B, Layer 22/26 in Llama-3B, Layer 22 in Qwen-1.5B).
- This MLP block receives a residual stream state that correctly encodes the count (or a state close to it) but overwrites it with a fixed, incorrect integer (e.g., "8" or "14").
- The MLP input is count-invariant; it fires regardless of the actual sequence length (e.g., for $n=8$ to $n=15$ ), indicating it responds to a specific format signature rather than performing a calculation.
- Causal ablation of this MLP block does not restore the correct answer; instead, it shifts the failure to a different wrong value (e.g., from "8" to "16"), suggesting the prior is partially distributed across multiple blocks.
Format and Token Specificity:
- Format Sensitivity: In larger models (Llama-3B, Qwen-3B), changing the delimiter from spaces to commas suppresses the MLP trigger, allowing the correct count to pass through. In smaller models (Llama-1B), the prior is format-invariant.
- Token Type: The prior fires for repeated alphabetic tokens (words) but not for repeated digit tokens. Models that fail to count repeated words correctly count repeated digits (e.g., "1", "1", "1") without error, proving the failure is a learned association between the "repeated word-list" format and a specific wrong output, not a general inability to estimate sequence length.
Anomaly Detection vs. Counting:
Larger models (Qwen-3B, 7B) that solve the P1 counting task still fail the P2 anomaly detection task (counting 9 apples and 1 banana). Mechanistic analysis shows that while attention mechanisms detect the anomaly (the banana receives high attention), this signal is not routed to the output to correct the count. This represents a second type of routing failure where the detection signal exists but is not converted into a behavioral correction.

Significance and Claims
The paper argues that counting failure in these models is a failure of routing, not a failure of representation. The internal circuitry correctly encodes the count, but a specific "prior" circuit intercepts and overwrites this information based on surface-level format cues (repeated word tokens).

The authors claim that conflating representation failures with routing failures leads to incorrect interventions (e.g., retraining for better counting capabilities when the capability already exists). Instead, interventions must target the specific circuits responsible for signal suppression. The study demonstrates that behavioral failures in capable models are often failures of expression, and identifying the specific routing pathway is a necessary prerequisite for any successful intervention. This work establishes a general phenomenon where internal truth and behavioral output can diverge completely, necessitating mechanistic analysis to distinguish between the two.