💻 computer science

Fragile Reasoning: A Mechanistic Analysis of LLM Sensitivity to Meaning-Preserving Perturbations

This paper reveals that large language models exhibit significant fragility to meaning-preserving perturbations in mathematical reasoning and introduces the Mechanistic Perturbation Diagnostics (MPD) framework to trace these failures to specific architectural patterns, categorizing them into localized, distributed, and entangled types with varying recoverability through targeted interventions.

Original authors: Shou-Tzu Han, Rodrigue Rizk, KC Santosh

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shou-Tzu Han, Rodrigue Rizk, KC Santosh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant student who can solve complex math problems with 90% accuracy. You ask them, "If Alice has 5 apples and buys 3 more, how many does she have?" They instantly say, "8."

But then, you ask the exact same question, just changing the name: "If Bob has 5 apples and buys 3 more, how many does he have?" Suddenly, the student panics and says, "12."

You try again, changing the format: "If Alice has five apples..." and they say, "15."

This is exactly what researchers found when they tested three popular AI models (Mistral, Llama-3, and Qwen). Despite being "smart," these models are surprisingly fragile. If you change the way a problem is written without changing the meaning, they often flip their answer from correct to wrong.

Here is a breakdown of the paper's findings using simple analogies.

1. The Problem: The "Glass House" of Reasoning

The researchers took 677 math problems and created "twins" for each one.

Name Swap: Changing "Alice" to "Bob."
Number Swap: Changing "$5" to "5 dollars."

The Result: The AI models failed 28% to 45% of the time on these "twins." It's like a car that drives perfectly on a straight road but crashes the moment you paint a different color on the steering wheel. The models aren't truly "understanding" the math; they are memorizing patterns.

2. The Investigation: The "Mechanic's Toolkit"

To figure out why the AI breaks, the authors built a diagnostic framework called MPD (Mechanistic Perturbation Diagnostics). Think of this as a mechanic taking apart an engine to see which specific gear is grinding.

They used four tools:

Logit Lens (The X-Ray): This lets them see what the AI is "thinking" at every single step of its processing. They found that for failed questions, the AI starts thinking the wrong answer much earlier in the process than for successful ones.
Activation Patching (The "Time Travel" Fix): Imagine the AI makes a mistake at step 10. The researchers took the "correct brain state" from a similar question and pasted it into the AI at step 10. If the AI suddenly got the answer right, the problem was localized to that specific step.
Component Ablation (The "Turn Off" Test): They turned off specific parts of the AI (like the "Attention" heads that focus on keywords, or the "MLP" layers that do the heavy lifting) to see which part was causing the error.
CAI (The "Domino Effect" Meter): This measures how much a small mistake grows as it travels through the AI's layers. Does a tiny error get amplified into a huge disaster, or does the AI self-correct?

3. The Discovery: Three Different "Personalities" of Failure

The most exciting finding is that the three AI models failed in three completely different ways. It's not just "AI is bad at math"; it's how they are bad.

A. The "Localizer" (Llama-3)

The Analogy: Imagine a house with a single, weak door. If you kick that specific door, the house falls. But if you fix that one door, the house is safe.
What happened: When Llama-3 failed, the error was trapped in a very specific layer (a specific step in its thinking).
The Fix: Because the problem was isolated, the researchers could "patch" it. By swapping in the correct brain state at that specific layer, they fixed 71% of the errors. It's highly repairable.

B. The "Distributed" (Mistral)

The Analogy: Imagine a house where the entire foundation is made of sand. If you push anywhere, the whole thing wobbles. There is no single weak point; the weakness is everywhere.
What happened: When Mistral failed, the error was spread out across many layers.
The Fix: Because the error was everywhere, you couldn't fix it by patching one spot. The researchers could only fix 5% of the errors. It's very hard to repair.

C. The "Entangled" (Qwen)

The Analogy: Imagine a knot of tangled headphones. You can't just pull one wire to fix it; the wires are so mixed up that pulling one tightens the whole knot.
What happened: In Qwen, the error wasn't just in the "focus" parts (Attention) but deeply mixed into the "calculation" parts (MLP). The components were so intertwined that fixing one broke another.
The Fix: This was the hardest to fix. The researchers managed to fix 0% of the errors using standard patches, and even trying to "surgically" edit the weights made things worse.

4. The "Domino Effect" (Cascading Amplification)

The researchers introduced a new metric called CAI (Cascading Amplification Index).

Stable AI: When you change the wording, the AI notices the difference, but it says, "Oh, that's just a different way of saying the same thing," and the error dies out.
Fragile AI: The AI notices the difference, gets confused, and that confusion gets louder and louder as it moves through its layers, until it screams the wrong answer.
Why it matters: This "Domino Effect" is a better predictor of failure than just looking at when the AI first got confused.

5. The Takeaway: "Repairing" the AI

The paper concludes that we can't just treat all AI failures the same.

If an AI is a Localizer (like Llama-3), we can fix it by targeting specific layers.
If it's Distributed or Entangled (like Mistral or Qwen), simple fixes won't work. We need to rethink how the model is built or trained.

In short: These AI models are like brilliant but fragile students. They can solve the math, but they get tripped up by the smallest change in the question. The researchers have now built a map to show us exactly where their brains break, which is the first step toward building AI that is truly robust and reliable.

1. Problem Statement

Despite achieving high accuracy on mathematical reasoning benchmarks like GSM8K, Large Language Models (LLMs) exhibit surprising fragility when faced with meaning-preserving surface perturbations. These are trivial modifications to the input (e.g., swapping character names or rephrasing numbers like "$5" to "5 dollars") that do not alter the mathematical logic or the correct answer.

The paper addresses the gap between benchmark performance and reasoning reliability. While prior work has documented that models fail under these conditions (behavioral level), this study investigates why and how these failures originate within the model's internal mechanics (mechanistic level). The core question is whether these failures stem from brittle pattern matching rather than genuine mathematical understanding.

2. Methodology

The authors propose the Mechanistic Perturbation Diagnostics (MPD) framework, a unified diagnostic pipeline combining four key techniques.

A. Dataset Construction

Source: 677 valid problem pairs from the GSM8K test set.
Perturbations: Two types of meaning-preserving transformations were applied:
1. Name Substitution: Replacing character names with random alternatives (341 pairs).
2. Number Format Paraphrasing: Rephrasing numerical expressions (e.g., " $5"$ \to$ "5 dollars") (336 pairs).
Evaluation: Models were tested using greedy decoding. A "flip" is defined as a case where the model answers the original correctly but fails on the perturbed variant.

B. Mechanistic Analysis Techniques

The study applies three complementary interpretability tools to a subset of 60 flipped samples per model:

Logit Lens Analysis: Projects hidden states at each layer through the unembedding matrix to identify the First Divergence Layer (the earliest layer where the prediction for the original and perturbed inputs differs).
Activation Patching: Replaces hidden states of the perturbed input with those from the correct original forward pass at specific layers. High recovery rates indicate localized failures; low rates indicate distributed sensitivity.
Component Ablation: Zeroes out the output of either Attention or MLP sublayers to determine which component causally contributes to the failure.
Cascading Amplification Index (CAI): A novel metric quantifying whether layer-wise divergence amplifies or attenuates across the network. It measures the ratio of divergence growth between layers ( $\Delta_{l+1} > \Delta_l$ ).

3. Key Contributions

Mechanistic Failure Taxonomy: The authors propose a formal classification of reasoning failures into three distinct types:
- Localized: Failures recoverable by patching specific layers (e.g., Llama-3).
- Distributed: Failures spread across redundant pathways, making them hard to localize (e.g., Mistral).
- Entangled: Failures where MLP sublayers dominate the causal mechanism, resisting standard attention-based interventions (e.g., Qwen).
Cascading Amplification Index (CAI): Introduced as a quantitative predictor of failure that outperforms the "First Divergence Layer" metric for architectures with distributed or localized failure modes (AUC up to 0.679).
Cross-Model Comparative Analysis: Demonstrated that models with similar parameter counts and training paradigms (7B–8B) exhibit qualitatively different internal failure signatures.
Repair Validation: Validated the taxonomy by showing that targeted interventions (steering vectors, layer fine-tuning) are significantly more effective on "Localized" failures than on "Distributed" or "Entangled" ones.

4. Key Results

A. Perturbation Sensitivity

All three models (Mistral-7B, Llama-3-8B, Qwen2.5-7B) showed substantial answer-flip rates (28.8% – 45.1%).

Mistral-7B: Highest flip rate (45.1%).
Llama-3-8B: Moderate flip rate (33.5%).
Qwen2.5-7B: Lowest flip rate (28.8%).
Observation: Number format paraphrasing was consistently more disruptive than name substitution across all models.

B. Mechanistic Signatures

Divergence Timing: Flipped samples diverge at significantly earlier layers than stable samples ( $p < 0.05$ $p < 0.05$ ).
- Mistral: Layer 9.1
- Llama-3: Layer 13.0
- Qwen: Layer 17.2
Activation Patching Recovery:
- Llama-3: Highly recoverable (71.7% of flips fixed by patching). Indicates Localized failure.
- Mistral: Low recovery (5.0%). Indicates Distributed failure.
- Qwen: Zero recovery (0.0%). Indicates Entangled failure.
Component Dominance:
- Mistral & Llama-3: Attention heads are the primary causal components.
- Qwen: MLP sublayers are the primary causal components (reversed pattern).

C. Repair Experiments

The study tested if the taxonomy predicts repair difficulty:

Localized (Llama-3): Steering vectors and layer fine-tuning recovered 12.2% and 11.3% of failures, respectively.
Entangled (Qwen): Steering vectors recovered only 7.2%; ROME editing (weight surgery) resulted in negative recovery (-1.1%).
Distributed (Mistral): Steering vectors recovered only 5.2%.

5. Significance and Conclusion

This paper fundamentally shifts the understanding of LLM reasoning fragility from a behavioral observation to a mechanistic diagnosis.

Architectural Specificity: Reasoning fragility is not a monolithic property; it manifests differently depending on the model architecture. Some models fail due to narrow bottlenecks (repairable), while others fail due to redundant or entangled pathways (hard to repair).
Predictive Power: The Cascading Amplification Index (CAI) provides a robust early-warning signal for potential reasoning failures, outperforming traditional divergence metrics.
Actionable Taxonomy: The proposed taxonomy (Localized, Distributed, Entangled) successfully predicts the efficacy of repair strategies. This suggests that a "one-size-fits-all" approach to improving reasoning robustness is insufficient; interventions must be tailored to the specific mechanistic failure mode of the model.

The findings imply that current high benchmark scores may mask significant underlying brittleness, and that true reasoning robustness requires addressing these specific internal failure signatures rather than just increasing training data or parameter count.