💬 NLP

Model Internal Sleuthing: Finding Lexical Identity and Inflectional Features in Modern Language Models

This paper systematically probes 25 modern language models across six languages to reveal that while inflectional features remain linearly decodable throughout the network, lexical identity representations weaken with depth as models trade specific word identity for compact, predictive structures.

Original authors: Michael Li, Nishant Subramani

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Michael Li, Nishant Subramani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a modern Large Language Model (LLM) not as a giant, scary robot brain, but as a multi-story skyscraper where information travels from the ground floor up to the penthouse.

This paper, titled "Model Internal Sleuthing," is like a team of detectives (the researchers) going into that skyscraper to see how it organizes its thoughts. They wanted to answer two specific questions:

Lexical Identity: Does the building remember what a word is (e.g., "walk")?
Inflectional Features: Does the building remember the grammar of the word (e.g., "walked" = past tense, "walking" = continuous)?

They tested 25 different "buildings" (models ranging from older ones like BERT to newer giants like Llama-3 and Qwen) across six different languages (English, Chinese, German, French, Russian, and Turkish).

Here is what they found, explained with some everyday analogies:

1. The "Name Tag" vs. The "Uniform"

Think of Lexical Identity (the word "walk") as a person's Name Tag, and Inflection (the "-ed" in "walked") as their Uniform.

The Finding: When a word enters the building (the first few floors), the Name Tag is very bright and clear. The model knows exactly what word it is.
The Twist: As the information travels up to the higher floors (deeper layers), the Name Tag starts to fade. By the time it reaches the top, the model cares less about the specific word "walk" and more about the context of the sentence. It's like the person taking off their name tag to blend in with the crowd.
The Uniform, however, never fades. The grammar (past tense, plural, etc.) stays visible and easy to read on every single floor, from the lobby to the penthouse. The model keeps the "Uniform" information locked in a very stable, easy-to-access pocket the whole way up.

2. The "Linear Elevator" vs. The "Maze"

The researchers tried to "read" the model's thoughts using two tools: a simple Linear Elevator (a straight path) and a complex Maze (a non-linear path).

Grammar (The Uniform): You can use the Linear Elevator to find the grammar information on any floor. It's a straight shot. The model keeps grammar organized in a neat, straight line.
Word Meaning (The Name Tag): On the lower floors, the elevator works fine. But on the upper floors, the elevator stops working. To find the specific word meaning up there, you have to navigate a Maze. The information is still there, but it's twisted and tangled up with other context.

3. The "Compression Room"

Some of the newer, faster models (like Qwen and Pythia) have a weird feature in the middle of the building: a Compression Room.

Imagine a hallway that suddenly squeezes down to a tiny tunnel before expanding again.
In these models, the middle layers squash the data so hard that the "dimensions" collapse. It's like trying to fit a whole library into a shoebox and then pulling it back out.
The Result: Even though the model can still guess the grammar correctly in this room, it becomes much harder to steer or control the model's behavior there. It's like trying to turn a car while the wheels are locked in mud; the engine runs, but the steering doesn't work as well.

4. The "Construction Site" (Pre-training)

The researchers also watched the building being built (during the training phase).

Grammar is built first. The "Uniform" (grammar rules) is constructed very early in the process. Once the foundation is laid, the grammar stays solid and doesn't change much.
Word Meaning is built last. The "Name Tags" keep changing and evolving as the building gets taller. The model spends a lot of time refining what the words mean in different contexts, often sacrificing the raw identity of the word to make better predictions about what comes next.

5. The "Language Difficulty" Factor

They tested this in six languages.

English, German, French, Russian: The model handled these well.
Turkish: This language is like a Lego set with infinite pieces (it's highly "agglutinative," meaning you can stick many grammar pieces onto one word). The model struggled the most here. The "Name Tag" faded very quickly, and the "Uniform" was harder to keep straight. It's like trying to organize a messy pile of Legos; the more complex the pieces, the harder it is for the model to keep them sorted.

The Big Takeaway

Modern AI models are incredibly smart, but they have a specific trade-off:

They keep grammar (inflection) rigid and accessible because it's the skeleton of language.
They let go of specific word identities (lexical) as they go deeper, because they need to focus on the bigger picture (context and prediction).

In short: If you ask a modern AI, "What is the past tense of 'run'?" it will tell you instantly, no matter how deep you look. But if you ask, "What specific word 'run' are we talking about right now?" the answer gets fuzzier the deeper you go, because the model is busy thinking about the story, not just the word.

1. Problem Statement

While Large Language Models (LLMs) dominate modern NLP, the internal mechanisms by which they encode linguistic information remain poorly understood, particularly in models newer than BERT and GPT-2. Previous research established a hierarchical organization in early models (e.g., lower layers for morphology, upper for semantics), but it is unclear if these patterns hold for modern architectures (decoder-only, encoder-decoder), varying scales (100M to 8B+ parameters), and diverse training regimes.

The authors specifically investigate two distinct linguistic properties:

Lexical Identity: The specific word form or lemma (e.g., distinguishing "walk" from "jump").
Inflectional Features: Grammatical attributes like tense, number, or case (e.g., distinguishing "walked" from "walks").

The core research question is: Where and how do modern LMs encode these properties, and how does this vary across model scale, architecture, and language typology?

2. Methodology

The authors conducted a systematic probing study across 25 pretrained models (ranging from BERT Base to Qwen2.5-7B and Llama-3.1-8B) and six typologically diverse languages (English, Chinese, German, French, Russian, Turkish).

A. Probing Setup

Data: Universal Dependencies corpora were used to extract target words and their associated lemmas and inflectional tags.
Representation Extraction: Hidden state activations were extracted from the residual stream at the position of the target word (using the last subword for tokenized inputs).
Probes: Two types of classifiers were trained at every layer:
1. Linear Regression Probe: To measure linear separability.
2. MLP Probe (Non-linear): A 2-layer perceptron to test if non-linear boundaries improve decoding.
Metrics:
- Selectivity: Defined as $Acc_{real} - Acc_{control}$ , where the control task uses random labels. This ensures the probe extracts genuine linguistic structure rather than memorizing surface correlations.
- Linear Separability Gap: The difference in selectivity between the MLP and Linear probes. A negative gap suggests the MLP is capturing spurious correlations rather than structural improvements.

B. Additional Analyses

Geometry: Measured Linear Effective Dimensionality using PCA to see how many components are needed to explain variance across layers.
Component Comparison: Compared Residual Stream activations vs. Attention Head outputs.
Steering: Applied steering vectors (difference of means between inflectional categories) to test causal controllability.
Pretraining Dynamics: Analyzed intermediate checkpoints of OLMo-2 and Pythia to track representation evolution during training.

3. Key Contributions

Systematic Cross-Architecture Analysis: The first large-scale study comparing lexical and inflectional encoding across 25 modern models (encoders, decoders, encoder-decoders) and six languages.
Divergent Representational Trajectories: Discovery that lexical identity and inflectional features follow fundamentally different paths through the network depth.
Geometry and Steering Correlation: Linking aggressive mid-layer dimensionality compression to reduced steering effectiveness, despite high probe accuracy.
Training Dynamics: Evidence that inflectional structures stabilize early in pretraining, while lexical identity representations continue to evolve and become more abstract (and less linearly decodable) as training progresses.

4. Key Results

A. Lexical Identity vs. Inflectional Features

Lexical Identity:
- Early Layers: Highly accessible and linearly decodable (high accuracy).
- Deep Layers: Accuracy drops significantly, and the information becomes increasingly non-linear.
- Selectivity: Remains near zero across depths, suggesting early high accuracy is driven by surface correlations rather than robust structural encoding.
Inflectional Features:
- Consistency: Remains linearly decodable and highly accurate (0.9–1.0) across all layers and architectures.
- Selectivity: Consistently positive (0.4–0.6), indicating robust, structural encoding.
- Language Variation: Encoding strength varies by morphological typology, with Turkish (highly agglutinative) showing the sharpest decline in lexical accuracy but maintaining strong inflectional linearity.

B. Representation Geometry & Steering

Dimensionality Compression: Models like GPT-2, Qwen2.5, and Pythia exhibit a "mid-layer collapse" where effective dimensionality drops drastically (few PCA components explain most variance).
Steering Effectiveness: In models with mid-layer collapse, steering effectiveness (the ability to change inflection via activation intervention) drops in those specific layers, even though probe accuracy remains high. This suggests that while the information is present, it is compressed into a way that makes it harder to manipulate causally.
Outlier Activations: The dimensionality collapse coincides with "massive activations" (outlier dimensions with unusually large magnitudes) in the middle layers.

C. Residual Stream vs. Attention Heads

Residual Streams: Contain significantly more linguistic information (both lexical and inflectional) than attention head outputs.
Attention Heads: Show lower accuracy and selectivity, particularly for lexical identity in middle layers. This supports the view that attention focuses on contextual aggregation, while the residual stream preserves token-level morphological details.

D. Pretraining Dynamics

Inflection: Stabilizes very early in training; accuracy and selectivity reach near-final values quickly.
Lexical Identity: Shows a trade-off. Early checkpoints have high linear accuracy, but as training progresses, the model abstracts away from rigid surface identities. Linear accuracy drops, but selectivity (distinguishing true structure from noise) increases, suggesting a shift toward more contextual, non-linear representations.

5. Significance and Implications

Robustness of Morphology: Despite rapid advances in model scale and architecture, transformers converge on a robust internal representation for core morphological properties (inflection). These features are linearly accessible and stable, unlike lexical identity.
Interpretability Limits: The finding that linear probes often have higher selectivity than MLP probes (negative linear separability gap) suggests that adding non-linear capacity often leads to overfitting or memorization of spurious correlations rather than uncovering deeper structure.
Mechanistic Control: The study highlights that high probe accuracy does not guarantee steerability. Models with aggressive dimensionality compression may encode information in a way that is "readable" but difficult to "edit" via simple activation interventions.
Future Directions: The results suggest that for tasks requiring morphological control, residual streams are the primary target for intervention, and that early training stages are critical for establishing grammatical structure.

In summary, the paper reveals that modern LMs treat inflection as a stable, linear, and steerable feature maintained throughout the network, while lexical identity is a transient, early-stage feature that becomes entangled and non-linear as the model builds contextual representations.