💬 NLP

When Probing Accuracy Saturates, Fragility Resolves: A Complementary Metric for LLM Pre-Training Analysis

This paper introduces "fragility," a complementary metric that measures the activation-noise level at which linear probe accuracy collapses, to reveal evolving representational structures and moral encoding gradients in LLM pre-training that remain invisible once standard probe accuracy saturates.

Original authors: Orion Reblitz-Richardson

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Orion Reblitz-Richardson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Satisfied Student"

Imagine you are a teacher trying to track how well a student is learning a subject over a whole school year. You give them a simple quiz every week.

Week 1: The student gets 50% right.
Week 2: They get 95% right.
Week 3 to Week 40: They keep getting 95% right.

If you only look at the score (accuracy), you would think the student stopped learning after Week 2. You'd say, "Great, they mastered it!" and stop paying attention.

But the student is actually still learning. They are refining their understanding, connecting ideas, and building a deeper, more robust knowledge base. The simple quiz just isn't hard enough to show the difference between "good enough" and "excellent."

This is exactly what happens with AI models.
When researchers use a standard test called "linear probing" to see if an AI understands a concept (like morality), the AI's score hits a ceiling (about 95%) very early in its training. For the remaining 95% of its training time, the score stays flat. The standard test becomes blind to all the complex changes happening inside the model.

The New Tool: The "Stress Test" (Fragility)

The authors of this paper say: "Let's stop just looking at the score. Let's see how sturdy the knowledge is."

They introduce a new metric called Fragility. Instead of just asking, "Can the AI answer this?" they ask, "How much noise can the AI handle before it gets confused?"

The Analogy: Imagine two people holding a heavy box.
- Person A is holding it perfectly still. If you tap their shoulder, they drop it. (High accuracy, but fragile).
- Person B is holding it with a wide, strong grip, maybe even using both hands and bracing their legs. If you tap their shoulder, they don't even wobble. (High accuracy, and robust).

Both people are holding the box (both have high accuracy), but Person B has a much deeper, more stable understanding of how to hold it.

In the paper, they add "noise" (random static) to the AI's brain while it answers. They measure how much noise it takes before the AI starts making mistakes.

Low Fragility (High Noise Tolerance): The AI is robust. It understands the concept deeply and redundantly.
High Fragility (Low Noise Tolerance): The AI is brittle. It's just memorizing a trick or a specific word, and a tiny bit of confusion breaks it.

What They Discovered

By using this "Stress Test" instead of just the "Score," the authors found three major things that were previously invisible:

1. The Learning Order: Words First, Then Meaning

They tracked how the AI learned about "morality."

The Old View: The AI learns morality very quickly (around step 1,000).
The New View: The AI actually learns in two stages.
1. Stage 1 (Step 1,000): It learns vocabulary. It knows that the word "betray" is bad and "greet" is good. This is easy to spot with a simple score.
2. Stage 2 (Step 5,000): It learns composition. It understands that the same word can be good or bad depending on the context (e.g., "hiding a secret to protect a sibling" vs. "hiding a secret to hurt a sibling").

The "Stress Test" showed that while the AI knew the words early on, it took much longer to build the sturdy, contextual understanding of why those words mattered.

2. The "Brittle Early Layers"

As the AI trained for thousands of steps, the "Stress Test" revealed a pattern in its brain structure:

The top layers of the AI became incredibly strong and stable (like a deep foundation).
The bottom layers became surprisingly fragile and brittle, even though the overall score was still high.

It's like a skyscraper where the top floors are reinforced with steel, but the bottom floors are made of cardboard. If you shake the building (add noise), the bottom might crumble even if the top looks fine. The standard score never saw this; the "Stress Test" did.

3. How You Teach Matters (Even if the Score Doesn't Show It)

The authors taught the AI three different ways:

Stories: Moral lessons inside fables (like Aesop).
Declarative: Repeating simple sentences like "Stealing is wrong" over and over.
Control: General text with no moral content.

The Result: All three groups got the exact same score on the final test.
The Difference:

The Story group built a sturdy, robust understanding.
The Declarative group (repeating "Stealing is wrong") built a fragile understanding. They memorized the pattern perfectly, but if you changed the sentence slightly, they broke.

This proves that how you curate data changes the internal "muscle" of the AI, even if the final test score looks identical.

The Bottom Line

The paper argues that accuracy is a bad ruler for measuring progress once a model gets "good enough." It hits a ceiling and stops telling us anything new.

By adding a Fragility Test (checking how much noise the model can handle), researchers can see the hidden structure of learning:

When concepts move from simple word-matching to deep understanding.
Which parts of the AI's brain are strong and which are weak.
Whether the AI truly "gets it" or just memorized a pattern.

In short: Don't just ask if the AI got the answer right. Ask how hard it would be to trick it. That is where the real story of learning is hidden.

Technical Summary: "When Probing Accuracy Saturates, Fragility Resolves"

1. The Problem: The Saturation of Probing Accuracy

Standard linear probing is a widely used interpretability protocol where a linear classifier is trained on frozen hidden states to predict a specific property (e.g., sentiment, morality, syntax). If the classifier achieves high accuracy, the property is declared "encoded."

The paper identifies a critical failure mode in this protocol when applied to pre-training dynamics. On the OLMo-2 1B trajectory, standard probing accuracy for properties like morality saturates within the first few thousand steps (reaching ~~95% accuracy by step 4K). For the remaining ~95% of the training trajectory (~~33K steps), the accuracy metric remains flat. Consequently, the instrument becomes blind to the continued representational changes occurring within the model, rendering the vast majority of pre-training "invisible" to standard analysis.

2. Methodology: Introducing "Fragility"

To recover the resolution lost to saturation, the authors introduce fragility, a complementary per-layer metric.

Definition: Fragility is defined as the critical noise level ( $\sigma^*_\ell$ ) at which a trained probe's accuracy collapses below a specific threshold (set to 0.6, or chance + 0.1).
Procedure:
1. Train a standard linear probe on clean hidden states for a specific layer $\ell$ .
2. Inject Gaussian noise ( $\epsilon \sim \mathcal{N}(0, \sigma^2 I)$ ) into the cached test-set activations.
3. Re-evaluate the probe across a logarithmic sweep of noise scales ( $\sigma \in \{0.1, 0.3, 1.0, 3.0, 10.0\}$ ).
4. Identify the smallest $\sigma$ where accuracy drops below the threshold.
Interpretation:
- Low $\sigma^*_\ell$ : The representation is fragile; the decision boundary has a narrow margin or low redundancy, causing accuracy to collapse under small perturbations.
- High $\sigma^*_\ell$ : The representation is robust; the distinction is encoded with a wide margin and/or high redundancy.
Scope: This metric is applied to the same trained probes used for accuracy curves, making it a direct complement that tracks the geometry of separability and redundancy even after accuracy plateaus.

3. Key Contributions and Experimental Setup

The paper validates this methodology using the OLMo-2 1B (37 checkpoints, dense 1K-step sampling) and OLMo-3 7B (20 checkpoints) families. The authors employ four distinct minimal-pair datasets to probe different capabilities:

Standard Moral: Single-token swaps of morally loaded lexemes (e.g., "betrayed" vs. "greeted").
Sentiment: Single-token swaps of adjectives.
Syntax: Structural well-formedness violations.
Compositional Moral (New Contribution): A novel dataset of 200 pairs where moral valence is determined by the integration of multiple tokens (e.g., "protect" vs. "humiliate" in specific contexts) rather than a single strong lexeme. This dataset is rigorously validated to ensure it cannot be solved by bag-of-words statistics alone (requiring multi-token integration).

4. Key Results

Finding 1: A Quantitative Lexical $\to$ Compositional Gradient
Standard probing accuracy suggests moral encoding emerges at step 1K. However, the authors demonstrate this measures lexical separability, not compositional reasoning.

Standard Moral Probe: Onsets at step 1K (single lexeme swap).
Compositional Moral Probe: Onsets at step 5K (multi-token integration).
Gradient: The emergence follows a strict order: Lexical moral detection (1K) $\to$ Sentiment (2K) $\to$ Compositional moral encoding (5K) $\to$ Syntax (6K).
Validation: The compositional probe transfers across construction types (e.g., training on "motive" pairs, testing on "role" pairs) and significantly outperforms a bag-of-words baseline, confirming it captures genuine compositional encoding, not just vocabulary lookup.

Finding 2: Fragility Resolves Structure Where Accuracy is Flat
While mean probing accuracy plateaus at ~0.95 by step 4K and remains static, mean critical noise (fragility) continues to evolve through step 36K.

Layer-Depth Gradient: A monotonic gradient develops where late layers maintain maximum robustness (high $\sigma^*$ ), while early layers become progressively more brittle (critical noise drops from 10.0 to 1.8).
Replication: This pattern holds for the OLMo-3 7B model and is robust across four random seeds for the compositional probe.
Significance: The model undergoes significant representational refinement (changing robustness and redundancy) long after it can simply "classify" the data correctly.

Finding 3: Data Curation Reshapes Robustness Without Changing Accuracy
The authors fine-tuned the OLMo-2 1B model on three matched corpora (Narrative Moral, Declarative Moral, General Control) using LoRA.

Accuracy: All three conditions produced identical peak probing accuracy (~0.74–0.75).
Fragility: The conditions produced distinct fragility fingerprints.
- Declarative Moral ("Stealing is wrong" templates) resulted in diffuse fragility (10 of 16 layers were fragile; mean $\sigma^* = 5.63$ ).
- Narrative/General conditions were more robust (mean $\sigma^* \approx 7.0$ ).
Implication: Memorizing syntactic templates creates narrow, low-margin decision boundaries that collapse under noise, a structural change invisible to accuracy metrics.

5. Significance and Claims

The paper makes a methodological claim rather than a specific moral-domain claim: Probing accuracy is a thresholded, capped metric that fails to capture dynamics once a property is linearly decodable.

The Core Thesis: In every comparison tested, where probing accuracy returns a flat, uninformative answer, fragility returns a structured, evolving one.
Reframing Emergence: The paper argues that claims of "emergence" based solely on standard probe onset (e.g., "morality emerges at step 1K") are often artifacts of lexical accessibility. True compositional capabilities emerge later and follow a different dynamic (gradual vs. step-like).
Utility: Fragility serves as a necessary diagnostic for pre-training analysis, revealing the evolution of representation geometry (margin and redundancy) that standard accuracy obscures.

The authors conclude that the pre-training window performs significantly more work than standard interpretability instruments reveal, and adding a fragility readout is essential to recovering this missing resolution.