💬 NLP

From Early Encoding to Late Suppression: Interpreting LLMs on Character Counting Tasks

This paper reveals that large language models often fail at character counting tasks not due to a lack of internal representation, but because specific negative circuits in later layers actively suppress correct signals in favor of incorrect, higher-probability outputs.

Original authors: Ayan Datta, Mounika Marreddy, Alexander Mehler, Zhixue Zhao, Radhika Mamidi

Published 2026-04-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ayan Datta, Mounika Marreddy, Alexander Mehler, Zhixue Zhao, Radhika Mamidi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant, super-smart librarian who has read almost every book in the world. They can write poetry, solve complex riddles, and summarize history books better than anyone else. But, if you ask them a simple question like, "How many times does the letter 'p' appear in the word 'apple'?", they might confidently tell you the answer is "1" or "2," even though the correct answer is "2."

This paper investigates why these super-smart AI models (Large Language Models or LLMs) fail at such simple tasks. The researchers discovered something surprising: The AI actually knows the answer, but it chooses to ignore it.

Here is the breakdown of their findings using simple analogies:

1. The "Hidden Note" Analogy (Early Encoding)

Think of the AI model as a factory assembly line with many stations (layers).

The Discovery: When the AI reads the word "apple," the very first stations on the line immediately write down a "sticky note" that says, "Hey, there are two 'p's here."
The Proof: The researchers used a tool called a "probe" (like a metal detector) to scan the AI's internal memory at different stages. They found that this "sticky note" with the correct count was clearly visible and readable in the early and middle stations of the factory. The information was there, loud and clear.

2. The "Censor" Analogy (Late Suppression)

So, if the AI knows the answer, why does it give the wrong one?

The Discovery: As the information travels down the assembly line toward the final output (the answer the user sees), it hits a specific group of workers in the last few stations. These workers act like censors or editors.
The Mechanism: Instead of passing the correct note ("2") to the final speaker, these late-stage workers actively crumple up the correct note and replace it with a simpler, more "comfortable" guess.
- Some models (like Qwen) act like a stubborn child who always guesses "1" because that's what they've seen most often.
- Other models (like LLaMA) act like a confused person who just picks a random number because they don't trust their own memory.
The Result: The final answer is wrong, not because the factory didn't know the answer, but because the final editors decided to suppress the truth in favor of a "safer" or more probable guess.

3. The "Broken Compass" Analogy (No Specialized Routing)

You might think the AI has a special "letter-counting team" (like a dedicated department) that handles this task.

The Discovery: The researchers looked for this team and found nothing. There is no specific "letter-counting department."
The Reality: The information about the letters is scattered across many different workers in a messy, uncoordinated way. It's like trying to pass a message through a crowded room where everyone is shouting different things. By the time the message reaches the end, it gets lost or overwritten by the loudest voices (the incorrect guesses).

4. The "Bigger Isn't Better" Twist

Usually, when we make AI models bigger (adding more parameters, like giving the factory more workers), we expect them to get smarter.

The Finding: The researchers tested bigger models (up to 9 billion parameters) and even models that had been "trained" to follow instructions better.
The Result: They failed just as badly. Making the factory bigger didn't fix the problem. The "censors" at the end of the line were still there, still crumpling the correct notes. In fact, training the models to be more obedient sometimes made them worse at counting because they learned to rely even more on their bad habits (like always guessing "1").

The Big Takeaway

This paper tells us that the problem isn't that AI is "dumb" or lacks the capacity to understand letters. The problem is structural.

Imagine a car with a perfect engine and a perfect GPS, but the driver (the final layer of the AI) has a habit of ignoring the GPS and driving in circles because they think they know a shortcut. The car has the right map, but the driver refuses to use it.

Why does this matter?
If these models can't be trusted to count letters—a task a 5-year-old can do—it suggests they might be fragile in other areas too. They might be "hallucinating" (making things up) not because they don't know the facts, but because their internal "censors" are actively suppressing the truth to give a more confident-sounding, but incorrect, answer.

In short: The AI isn't failing because it can't see the answer; it's failing because it's been trained to ignore the answer it sees.

1. Problem Statement

Large Language Models (LLMs) exhibit a paradoxical failure mode: they excel at complex natural language understanding and reasoning benchmarks but consistently fail at elementary symbolic tasks, such as counting the occurrences of a specific character in a word (e.g., "How many 'p's are in 'apple'?").

The Gap: While prior studies have documented these failures, the internal mechanisms causing them remain underexplored. It is unclear whether models fail because they cannot encode the necessary sub-token information or because they fail to utilize that information during the generation process.
Hypothesis: The authors investigate whether the failure stems from a lack of representational capacity or from structured interference within the model's computation graph.

2. Methodology

The authors employ a controlled diagnostic framework using character counting as a minimal probe to isolate token-level reasoning from higher-level linguistic confounds.

A. Experimental Setup:

Models: Evaluated across three major model families (LLaMA 3.2, Qwen 2.5, Gemma 2) in both base and instruction-tuned variants, ranging from 2B to 9B parameters.
Dataset: Constructed from WordNet and Wiktionary, featuring 10,000 words with balanced target counts (1, 2, or 3). Multiple prompt templates were used to minimize phrasing bias.
Metrics:
- Logit Difference ( $\Delta$ ): Measures the model's preference for the correct token relative to alternatives in the residual stream.
- Probability Gap ( $\Delta_p$ ): Measures the suppression of the correct answer's probability relative to the model's actual prediction.

B. Interpretability Techniques:

Probing Classifiers: Linear classifiers trained on hidden states to determine if character-count information is linearly decodable at various layers (token-level, word-level, and prompt-level).
Logit Lens: Projects intermediate hidden states onto the vocabulary space to trace how predictions evolve and where correct signals are lost.
Activation Patching (Causal Tracing): Systematically overwrites hidden activations from a "corrupted" input (e.g., wrong letter/word) with "clean" activations to measure the causal contribution of specific components (layers, attention heads, MLPs) to the correct output.

3. Key Findings & Results

A. Information is Encoded but Not Used:

Early Encoding: Probing classifiers reveal that character-count information is reliably encoded in the residual stream of early and mid-layers (accuracy >80-90%). This holds true even for words the model has never seen during training.
Information Loss: While information is present at the word level, probe accuracy drops significantly at the final prompt token (the position used for prediction). This indicates a disconnect between representation and utilization.

B. The "Negative Circuit" (Late Suppression):

Suppression Mechanism: The primary cause of failure is not missing information but active suppression in the final layers.
Logit Lens Analysis: Shows that while the correct count is available in intermediate layers, the probability of the correct token decreases in the final layers, while the probability of incorrect, degenerate answers increases.
Specific Components:
- Qwen Models: Suppression is distributed across the penultimate-layer MLP, final-layer attention heads, and final-layer MLP.
- LLaMA Models: Suppression is almost entirely driven by the final-layer MLP.
- Attention Heads: No specific attention heads were found to be dedicated to routing character information; the signal is fragmented, and no single head shows strong causal restoration.

C. Degenerate Answer Strategies:

Models do not attempt to count; they fall back on simple heuristics:
- Qwen/Gemma: Overwhelmingly predict a fixed count (usually "1").
- LLaMA: Exhibits near-uniform random predictions across counts.
These strategies are reinforced by the final layers, overriding the correct intermediate signals.

D. Scaling and Instruction Tuning Do Not Help:

Scaling: Increasing model size from 3B to 9B parameters does not alleviate the failure. The suppression patterns remain consistent.
Instruction Tuning: Post-training (Instruct variants) shifts the specific degenerate strategy (e.g., alternating between "1" and "2") but does not fix the underlying mechanism. The models still suppress correct signals in favor of learned priors.

4. Key Contributions

Mechanistic Explanation of Failure: The paper demonstrates that symbolic reasoning failures in LLMs are not due to missing representations or insufficient scale, but rather structured interference where specific late-layer components (negative circuits) systematically downweight correct symbolic signals.
Competitive Decoding Evidence: It provides empirical evidence that LLM forward passes implement a form of competitive decoding, where correct and incorrect hypotheses coexist, and the final output is determined by the suppression of correct signals as much as by the amplification of incorrect ones.
Identification of Negative Circuits: The authors identify specific subnetworks (primarily final-layer MLPs and some attention heads) that act as "negative circuits," actively attenuating correct information in favor of higher-probability but incorrect outputs.

5. Significance and Implications

Robustness: The findings suggest that the robustness of LLMs on complex tasks may rest on unstable foundations, as they struggle with atomic operations required for rigorous reasoning.
Design Strategy: Current scaling laws and instruction tuning are insufficient to fix these structural weaknesses. Future architectures need design strategies that ensure information is not only encoded but also reliably propagated to the output layer without being suppressed by degenerate heuristics.
Interpretability: The study highlights the necessity of mechanistic interpretability tools (like activation patching and logit lens) to diagnose failures that surface-level accuracy metrics miss. It reveals that "knowing" (encoding) and "doing" (generating) are decoupled processes in current LLMs.

In conclusion, the paper argues that LLMs are capable of performing character counting internally but are "trained to forget" the correct answer in favor of simpler, statistically probable patterns during the final generation stage.