💬 NLP

Categorical Perception in Large Language Model Hidden States: Structural Warping at Digit-Count Boundaries

This paper demonstrates that large language models exhibit categorical perception-like geometric warping in their hidden states at digit-count boundaries (e.g., 10 and 100), a phenomenon driven by structural tokenization discontinuities rather than semantic knowledge, which manifests as either explicit categorization or purely geometric distortion depending on the model architecture.

Original authors: Jon-Paul Cacioli

Published 2026-03-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jon-Paul Cacioli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are looking at a long, smooth road. As you drive down it, the distance between mile markers is perfectly even. This is how a computer usually sees numbers: a smooth, continuous line where 9 is just a tiny bit smaller than 10, and 10 is just a tiny bit smaller than 11.

But this paper discovered something strange happening in the "mind" of modern AI (Large Language Models). When these AIs look at numbers, their internal map of the world suddenly warps at specific spots, just like a rubber sheet being stretched.

Here is the story of that discovery, broken down into simple concepts.

1. The "Rubber Sheet" Effect (Categorical Perception)

In human psychology, there's a famous idea called Categorical Perception. It means our brains treat things differently when they cross a "border."

The Analogy: Imagine a color wheel. Red and orange are right next to each other. But if you stand exactly on the line between them, your brain suddenly says, "Whoa, these are totally different colors!" even though the physical change is tiny. You can tell the difference between "Red" and "Orange" much better than you can tell the difference between two shades of Red.

The researchers found that AI models do this exact same thing with numbers, but for a very specific reason: The way the numbers are written.

2. The "Doorway" in the Code

The AI doesn't just see the value of a number; it sees the letters (tokens) that make up the number.

The Number 9: Written as "9". That's one character.
The Number 10: Written as "10". That's two characters.

To the AI, the jump from 9 to 10 isn't just a small step up in value. It's like walking through a doorway where the floor suddenly changes texture. The "doorway" is the moment a single-digit number becomes a double-digit number.

The study found that inside the AI's brain, the distance between 9 and 10 suddenly stretches out. It's as if the AI thinks 9 and 10 are much further apart than 8 and 9, even though mathematically they are the same distance. This happens again at 99 and 100 (two digits to three digits).

3. The Two Types of "AI Brains"

The researchers tested six different AI models and found two distinct types of reactions to this "doorway":

The "Classic" AI (Gemma, Qwen): These models act like a human. They not only have the stretched map (geometry), but they can also tell you the difference. If you ask, "Is this a single-digit or double-digit number?", they will say "Single" or "Double" with high confidence. They know the rule.
The "Structural" AI (Llama, Mistral, Phi): These models have the stretched map, but they can't tell you the rule. If you ask them the same question, they might guess randomly or get confused.
- The Metaphor: Imagine a person who has a perfect internal GPS that shows a huge canyon between 9 and 10. But if you ask them, "Is there a canyon here?" they shrug and say, "I don't know." They feel the difference in their "bones" (the math inside them), but they can't speak about it.

This is a huge discovery: You can have the geometric shape of a category without actually knowing the category exists.

4. The "Hot vs. Cold" Test

To prove this wasn't just about numbers, the researchers tested the AI with temperature.

We have words for "Hot" and "Cold." There is a boundary (around 22°C / 72°F) where things switch from cold to hot.
The Result: The AI did not stretch its map at this boundary. Why? Because the word "21" and "23" look the same to the AI's tokenizer (the part that breaks text into pieces). There is no "doorway" in the code.
The Lesson: The AI isn't stretching the map because it learned that 10 is special. It's stretching it because the format of the input changed (from one digit to two). The "shape" of the data forces the brain to warp.

5. The "Early vs. Late" Layers (The Hidden Work)

AI models have many layers of processing, like floors in a skyscraper.

The Top Floors: This is where the AI's internal map looks the most "stretched" and clear. It's the most legible.
The Basement: This is where the actual work happens.
The Twist: The researchers tried to "poke" the AI's brain at different floors. When they poked the top floors (where the map looked best), nothing happened. But when they poked the basement (early layers), the AI's behavior changed instantly.
The Metaphor: It's like looking at a finished painting. The painting looks perfect on the canvas (the top layer). But the artist didn't paint the final strokes there; they did the heavy lifting on the sketchpad in the basement. The "visible" part of the AI's brain isn't where the thinking is actually happening.

Summary: What Does This Mean?

Format is King: AI models create "categories" (like single-digit vs. double-digit) just because the way we write numbers changes format. They don't need to "learn" math to do this; the structure of the text forces their brain to warp.
Feeling vs. Knowing: An AI can have a deep, structural understanding of a boundary (feeling the stretch) without being able to explain it (knowing the rule).
Look Deeper: Just because a part of an AI's brain looks "smart" or "clear" doesn't mean it's doing the work. The real magic often happens in the messy, early layers.

In short, this paper shows that how we write things down changes how machines think about them, creating invisible "walls" in their minds that separate numbers, even if the machine doesn't realize it's doing so.

1. Problem Statement

Categorical Perception (CP) is a well-documented phenomenon in human psychology where stimuli varying continuously along a physical dimension are perceived as falling into discrete categories, resulting in enhanced discriminability at category boundaries. While CP has been studied extensively in speech, color, and facial expressions, it remains unclear whether this phenomenon exists in Large Language Models (LLMs) and, if so, whether it arises from learned semantic knowledge (acquired CP) or structural properties of the input format (structural CP).

The core challenge in biological systems is that representational geometry is not directly observable; it must be inferred from behavior. LLMs offer a unique testbed because their hidden-state representations are directly accessible. This study investigates whether LLMs exhibit the full geometric signature of CP (representational warping) when processing Arabic numerals at digit-count boundaries (e.g., 9→10, 99→100), and whether this geometric warping is causally linked to the model's ability to explicitly categorize or discriminate those numbers.

2. Methodology

The study employed a rigorous psychophysical framework applied to the hidden states of six LLMs across five architecture families (Llama-3, Mistral, Gemma, Qwen, Phi).

Models: Six models (3.8B to 9.2B parameters) including both instruction-tuned and base variants.
Stimuli:
- Numerical Domain: Probing values spanning the 9→10 (single-to-double digit) and 99→100 (double-to-triple digit) boundaries.
- Control Domains:
  - Non-boundary controls: Values centered at 15 and 150 (no digit-count transition).
  - Temperature Domain: Values spanning the linguistic "hot/cold" boundary (~22°C) but with no tokenization discontinuity (e.g., "21" and "23" tokenize identically).
  - Nonce-Token Control: Ordinal sequences of nonsense words ("glorp", "blicket") with and without explicit ordering context to test if ordinality alone induces CP.
Experimental Paradigms:
1. Representational Similarity Analysis (RSA - Paradigm A): Compared empirical Representational Dissimilarity Matrices (RDMs) of hidden states against five theoretical models: Continuous (Log-distance), CP-Additive (Log-distance + boundary boost), CP-Multiplicative, Categorical, and Linear.
2. Identification (Paradigm B0): Forced-choice tasks asking models to classify numbers as "small/large" or "single/multi-digit."
3. Behavioral Discrimination (Paradigm B): Forced-choice "Which is larger?" tasks measuring accuracy and decision confidence (via $|\Delta \text{logit}|$ ).
4. Precision Gradient (Paradigm C): Measured local representational precision ( $1/||h_{n+1} - h_n||$ ) to detect spikes at boundaries.
5. Causal Intervention (Paradigm E): Activation patching to test if manipulating the "category direction" in hidden states causally alters discrimination confidence.

3. Key Contributions

Discovery of Structural CP in LLMs: Demonstrated that LLMs exhibit the full geometric signature of categorical perception (representational warping) at digit-count boundaries, even in models that cannot explicitly identify the category.
Dissociation of Geometry and Behavior: Identified a fundamental split between "Classic CP" (models that both warp geometry and explicitly categorize, e.g., Gemma, Qwen) and "Structural CP" (models that warp geometry but fail explicit categorization, e.g., Llama, Mistral, Phi).
Proof of Structural Mechanism: Provided evidence that CP in LLMs is driven primarily by input-format discontinuities (tokenization, character count, lexical form changes) rather than learned semantic category knowledge.
Geometry-Function Dissociation: Showed via causal intervention that the layers with the most interpretable geometric structure (peak RSA fit) are causally inert, while the computational work driving behavior occurs in earlier layers where the geometry is still forming.

4. Key Results

A. Representational Geometry (RSA)

Universal Warping: The CP-Additive model (log-distance + a constant boundary boost) fit the empirical hidden-state geometry significantly better than a purely continuous model at 100% of primary layers across all six models.
Boundary Specificity: This effect was absent at non-boundary control positions (15, 150) and in the temperature domain.
Magnitude Scaling: The effect size was 3.9x to 12.7x larger at the 100-boundary (2→3 digits) compared to the 10-boundary (1→2 digits), correlating with the magnitude of the structural discontinuity.
Additive Mechanism: The boundary effect operates as a constant additive displacement in representational space, not a multiplicative scaling, consistent with Fisher information warping theories.

B. Identification vs. Geometry (The "Structural CP" Phenomenon)

Classic CP (Gemma, Qwen): These models showed sharp sigmoid identification functions (explicit categorization) and geometric warping.
Structural CP (Llama, Mistral, Phi, Base): These models exhibited identical geometric warping to the Classic CP models but failed to produce a crossover in identification tasks (flat or biased responses).
Conclusion: Geometric categorical structure exists independently of the model's ability to report that category.

C. Control Conditions

Temperature Domain: Despite having a linguistic "hot/cold" boundary, the lack of tokenization discontinuity resulted in no CP effect (Continuous model fit better). This proves linguistic category knowledge alone is insufficient.
Nonce-Tokens: Ordinal information alone produced a weak CP effect (3–10x smaller than real numbers). Real numbers produced effects 10–100x larger, confirming that structural discontinuities (tokenization/character count) are the primary amplifiers.
Base vs. Instruct: The Llama-3 Base model (pre-instruction tuning) showed the same CP geometry as its Instruct counterpart, proving the effect is present in pre-trained representations and not learned via instruction.

D. Causal Intervention (Activation Patching)

Layer Dissociation: Patching the "category direction" in early layers (5, 8) caused large, dose-dependent changes in discrimination confidence.
Peak Inertness: Patching in mid-to-late layers (16–27), where the geometric warping (RSA fit) is strongest, had negligible effects on behavior.
Implication: The most "legible" representation is not the one performing the computation.

5. Significance and Implications

For Cognitive Science: This study provides a computational existence proof that categorical perception can arise from representational format constraints without explicit category learning. It challenges the view that CP requires learned semantic categories, suggesting that structural discontinuities in input are sufficient to warp perceptual space.
For AI Interpretability:
- Geometry $\neq$ Function: The study warns against equating the layer with the clearest geometric signal (highest RSA fit) with the layer performing the relevant computation.
- Behavioral Competence is Not Required: A model can possess a robust categorical structure in its hidden states without being able to explicitly demonstrate it in a behavioral task.
Methodological Advance: The paper successfully adapts formal psychophysical tools (RSA, Signal Detection Theory, causal intervention) to LLMs, moving beyond simple "probing" to a rigorous analysis of representational geometry and its causal role.

Conclusion: The paper establishes that structural input-format discontinuities (tokenization boundaries, digit-count transitions) are sufficient to produce the full geometric signature of categorical perception in LLMs. This "Structural CP" is a universal architectural feature that exists independently of semantic category knowledge and can be dissociated from the model's explicit behavioral competence.