💬 NLP

Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarks

This paper introduces a contrastive, LRP-based attribution framework with cross-layer capabilities to systematically analyze Large Language Model failures on realistic benchmarks, revealing that while token-level attribution offers informative signals in specific cases, it is not universally applicable for all failure types.

Original authors: Rongyuan Tan, Jue Zhang, Zhuozhao Li, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rongyuan Tan, Jue Zhang, Zhuozhao Li, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but sometimes stubborn, robot assistant (a Large Language Model, or LLM). Sometimes, when you ask it a question, it gives you a wrong answer. Usually, when we try to figure out why it failed, we just look at the input and the output. It's like looking at a car that won't start and saying, "Well, the engine is off," without ever opening the hood to see if the spark plugs are missing or the battery is dead.

This paper is about opening the hood of these AI robots to see exactly what's happening inside their "brains" when they make mistakes.

Here is the breakdown of their work using simple analogies:

1. The Problem: The "Black Box" Mystery

Most previous studies looked at AI failures like a detective looking at a crime scene from the outside. They knew what happened (the AI gave a wrong answer), but they didn't know why.

The Old Way: "The AI said 'Paris' instead of 'London'. Maybe it's confused?"
The New Way: "Let's look inside the AI's brain to see which specific thoughts led it to choose 'Paris' and which thoughts it ignored."

2. The Tool: "Contrastive Attribution" (The "What-If" Game)

The authors created a special tool called Contrastive Attribution. Think of it as a "What-If" simulator.

When the AI makes a mistake (e.g., it picks the wrong word), the researchers ask: "What if the AI had picked the CORRECT word instead?"

They take the difference between the "Wrong Word" and the "Right Word."
They trace that difference backward through the AI's layers, like rewinding a video tape to see which frames caused the error.
They create a heat map (like a weather map showing hot and cold spots) that shows which words in your prompt were "hot" (influenced the wrong answer) and which were "cold" (ignored).

3. The Big Challenge: The "Long Story" Problem

AI models are great at reading long stories (thousands of words). But looking inside the brain of an AI reading a long story is like trying to watch a 10-hour movie frame-by-frame on a tiny calculator—it's too slow and crashes the computer.

The Innovation: The authors invented a "batching trick." Imagine instead of watching one movie frame at a time, you stack 100 frames together and process them all at once. This allowed them to analyze long, complex failures without the computer exploding.

4. What They Found: The "Three Types of Mistakes"

By looking at hundreds of failures, they found three main reasons why the AI gets it wrong:

Type A: The "Tunnel Vision" (Underweighting Relevant Tokens)
- Analogy: You ask the AI, "Write a poem without using the letter 'E'." The AI writes a poem full of 'E's.
- The Diagnosis: The heat map shows the AI completely ignored the instruction "without using the letter 'E'." It was looking at the wrong part of the prompt.
Type B: The "Distraction" (Overweighting Irrelevant Tokens)
- Analogy: You ask, "Who is the president of France?" The AI says, "The Eiffel Tower."
- The Diagnosis: The AI got distracted by the word "France" and the image of the Eiffel Tower in its training data, ignoring the word "President." It focused on the wrong clue.
Type C: The "Deep Brain Fog" (Hidden Internal Struggles)
- Analogy: Sometimes the AI gets the right clues, but somewhere deep inside its layers, it gets confused and flips a switch to the wrong answer.
- The Diagnosis: This is the hardest to see. The "heat map" of the input looks fine, but when they looked at the Attribution Graph (a map of how thoughts travel through the AI's layers), they saw that the "wrong" thought grew stronger as it traveled deeper into the brain, eventually overpowering the "right" thought.

5. The Good News: Bigger Brains Fix It (Mostly)

They tested this on small AI models and big AI models.

The Result: When they made the AI bigger (more "brain power"), it started getting the answers right.
The Proof: The heat maps showed that the bigger AI wasn't just guessing better; it was actually paying attention to the right clues. It stopped ignoring the instructions and stopped getting distracted by irrelevant words. This proves that making models bigger isn't just magic; it actually changes how they process information.

6. The Limitation: Not a Magic Wand

The paper is honest: this tool doesn't solve every mystery.

For simple mistakes (like ignoring an instruction), the tool works great.
For complex math problems, sometimes the tool can't see the error because the mistake happens in such a tiny, specific part of the brain (a single neuron) that the "heat map" is too blurry to see it. It's like trying to find a single lost needle in a haystack using a flashlight that only lights up the whole haystack.

Summary

This paper is like giving engineers a super-powered X-ray for AI brains.

Before: We knew the AI was sick, but we didn't know where.
Now: We can see exactly which "thoughts" (tokens) caused the sickness.
Why it matters: This helps developers fix AI models faster, understand why they fail, and build smarter, more reliable assistants for the real world.

The Bottom Line: We can finally stop guessing why AI fails and start seeing the actual mechanics of its mistakes, just like a mechanic finally getting to see the engine of a car that won't start.

1. Problem Statement

Despite the rapid advancement of Large Language Models (LLMs), they still exhibit significant failures in real-world scenarios (e.g., instruction following, reasoning, coding). Existing failure analysis methods primarily rely on behavioral analysis (examining inputs/outputs and perturbations), which identifies what fails but rarely explains why failures occur internally.

While interpretability tools (like Layer-wise Relevance Propagation or LRP) exist, their application has been limited to:

Toy settings: Short prompts or highly constrained templates.
Scalability issues: Inability to handle long-context inputs (thousands of tokens) common in realistic benchmarks like GAIA2 or agentic tasks.
Lack of contrast: Most methods explain why a model chose a token, but not why it chose an incorrect token over a correct alternative.

The paper addresses the gap: Can interpretability methods provide practical, actionable value for analyzing LLM failures on realistic, long-context benchmarks?

2. Methodology

The authors propose a framework centered on Token-Level Contrastive Attribution using an efficient extension of Layer-wise Relevance Propagation (LRP).

A. Contrastive Attribution Formulation

Instead of attributing the logit of a single token, the authors define the attribution target as the logit difference ( $\Delta \ell$ ) between an incorrect target token ( $t_{tgt}$ ) and a correct contrast token ( $t_{con}$ ):
$\Delta \ell = \ell(t_{tgt}) - \ell(t_{con})$
This formulation isolates the specific decision-making process that led to the error, removing shared, failure-irrelevant computations. The goal is to decompose $\Delta \ell$ back to input tokens and internal hidden states.

B. Efficient Attribution Graph Construction

Standard LRP requires propagating relevance through every layer, which is computationally prohibitive for long contexts if done naively (requiring $O(N)$ backward passes for $N$ tokens).

AttnLRP: The authors adopt AttnLRP (Attention-aware LRP), a state-of-the-art variant that offers high faithfulness for transformers.
Batch-Packed Multi-Target Backpropagation: To scale to long contexts, they introduce a batching trick. Instead of backpropagating relevance for each target token individually, they pack multiple attribution targets into a single batch dimension. This leverages GPU vectorization to compute relevance propagation between layers in $O(\lceil N/B \rceil)$ passes, making it feasible to construct cross-layer attribution graphs for inputs with thousands of tokens.
Coarse-to-Fine Analysis:
1. Coarse: Aggregate relevance at the hidden-state level to identify important subgraphs and input tokens.
2. Fine: Optionally refine analysis to the neuron level within identified subgraphs (though the paper focuses primarily on the coarse state-level for scalability).

C. Experimental Setup

Benchmarks: IFEval (Instruction Following), GAIA2 (Agentic/Long-context), MATH (Reasoning), and EvalPlus (Coding).
Models: Qwen3 series (0.6B, 1.7B, 4B, 8B) and Olmo-3-7B-Think (for training checkpoint evolution).
Data Collection: Identified "clean" failure cases where the model deviates from the correct trajectory. Used a semi-automated pipeline (LLM voting + human verification) to identify the target token (first error) and a contrast token (a plausible alternative that would have led to a correct trajectory).

3. Key Contributions

Formulation of Failure Analysis: Reframed LLM failure analysis as a token-level contrastive attribution problem, enabling the study of why models prefer incorrect tokens over correct alternatives in realistic settings.
Scalable Attribution Framework: Introduced an efficient batch-packed backpropagation method that enables the construction of cross-layer attribution graphs for long-context inputs, overcoming previous computational bottlenecks.
Systematic Empirical Study: Conducted the first large-scale interpretability analysis across diverse benchmarks, model sizes, and training checkpoints, characterizing attribution patterns in realistic failure scenarios.
Diagnostic Insights: Provided actionable findings on how attribution patterns shift with model scaling and training, offering a new lens for debugging and alignment.

4. Key Results

A. Attribution Outcomes & Failure Patterns

Explainability: In many cases (IFEval, GAIA2, EvalPlus), input attribution alone reveals failure signatures. However, for complex reasoning (MATH), input attribution is often insufficient, requiring attribution graph analysis to uncover internal dynamics.
Dominant Failure Modes:
- Underweighting Relevant Tokens (URT): The model fails to assign sufficient negative relevance to critical constraint tokens (e.g., "no commas").
- Overweighting Irrelevant Tokens (OIT): The model assigns high relevance to misleading or semantically uninformative tokens.
- Combined: Many failures involve both URT and OIT.
Graph Insights: Attribution graphs revealed that in some "unexplainable" cases, the error was driven by late-layer bias accumulation (e.g., the model's internal bias for a token like "First" overrode the correct token "«" despite early layers showing the correct preference).

B. Model Scaling Effects

Correction of Failures: Larger models (1.7B, 4B) consistently corrected the incorrect token preferences of smaller models (0.6B).
Interpretability Evidence: This correction was not just a surface-level change. Larger models showed:
- More negative relevance on instruction/constraint tokens (indicating better adherence to rules).
- Reduced reliance on superficial token continuation from previous answers.
- A shift in the attribution graph structure, where larger models suppressed the "attention sink" (BOS token) influence and better integrated context.

C. Training Evolution (Checkpoints)

Training Dynamics: Analyzing checkpoints of Olmo-3-7B-Think showed that failure correction is a gradual process.
Key Inflection Points: The most significant shifts in attribution patterns occurred during early Supervised Fine-Tuning (SFT), where the model learned to attend to task semantics and reduce reliance on superficial patterns.
RLVR Impact: Reinforcement Learning with Verifiable Rewards (RLVR) contributed to further refinement but induced smaller attribution shifts compared to SFT, likely because SFT had already corrected the primary failure modes.

D. Limitations

Math Reasoning: A non-trivial fraction of failures (especially in MATH) remained unexplained even with attribution graphs, suggesting that neuron-level analysis (beyond the current hidden-state aggregation) is required for complex numerical reasoning.
Not Universal: Contrastive attribution is not a "silver bullet"; it provides informative signals for specific classes of failures but cannot explain all model behaviors.

5. Significance and Practical Implications

Diagnostic Tool: The framework serves as a practical tool for developers to debug specific failure modes (e.g., identifying if a model is ignoring a constraint vs. hallucinating a token).
Targeted Prompt Tuning: Attribution heatmaps can identify the specific input tokens driving errors, allowing practitioners to refine prompts by masking or rephrasing those specific tokens.
Training Monitoring: Attribution patterns can track whether model improvements (via scaling or training) are due to genuine reasoning improvements or superficial memorization.
Alignment Training: The paper suggests using attribution scores as token-importance signals for alignment training (e.g., in DPO), focusing optimization on the specific tokens that drive the preference gap rather than treating all tokens equally.

Conclusion

The paper demonstrates that contrastive attribution is a viable and powerful method for analyzing LLM failures in realistic, long-context settings. By combining efficient graph construction with systematic empirical analysis, the authors reveal that model scaling and training lead to systematic, interpretable shifts in how models process information. While limitations remain for complex reasoning tasks, this work bridges the gap between theoretical interpretability and practical model debugging.