Applied Explainability for Large Language Models: A Comparative Study

This paper presents a comparative study of Integrated Gradients, Attention Rollout, and SHAP on a fine-tuned DistilBERT model for sentiment classification, revealing that gradient-based methods offer more stable and intuitive explanations while highlighting the trade-offs between computational efficiency, flexibility, and reliability in practical LLM explainability.

The Gist

Imagine you've hired a brilliant but mysterious chef (the Large Language Model) to cook a meal for you. This chef is amazing; they can make the most delicious dishes (sentiment analysis, answering questions, etc.) using a massive library of recipes. But here's the catch: the chef never tells you why they added a pinch of salt or a dash of pepper. They just say, "Trust me, it's good."

In the world of Artificial Intelligence, this is a big problem. If the chef serves you a dish that tastes like soap, you need to know why so you can fix it. This is where Explainability comes in. It's the art of peeking into the chef's mind to see what ingredients they thought were most important.

This paper is like a taste-test competition where the author tries three different "spy cameras" to see what the chef is actually thinking when they classify a movie review as "Good" or "Bad."

Here is the breakdown of the three cameras (methods) the author tested, using simple analogies:

1. The "Attention Rollout" Camera (The Glittery Spotlight)

• How it works: This method looks at the chef's eyes. In AI, "attention" is like the chef's gaze. If the chef looks at the word "wonderful" while cooking, this camera assumes that word is the most important.
• The Problem: The author found that the chef often looks at the plate or the spoon (structural words like "the," "a," or punctuation) just as much as the food ingredients.
• The Analogy: Imagine a detective trying to solve a crime by watching where the suspect's eyes darted. The suspect might look at a shiny watch on the table (distraction) rather than the weapon. This camera is fast and cheap, but it often gets distracted by shiny objects that don't actually matter to the final taste.

2. The "SHAP" Camera (The "What If?" Simulator)

• How it works: This method is a bit like a time-traveling simulator. It asks, "What if we removed the word 'wonderful'? Would the dish still taste good?" It tries to remove ingredients one by one to see how the final result changes.
• The Problem: It's incredibly accurate in theory, but it's slow and finicky.
• The Analogy: Imagine trying to figure out why a cake failed by baking 1,000 slightly different cakes, removing one ingredient at a time. It gives you a very precise answer, but by the time you're done, you've spent all your money on flour and eggs. Also, if you change the oven temperature slightly (input formatting), the results can jump around wildly. It's powerful but too expensive and unstable for everyday use.

3. The "Integrated Gradients" Camera (The Taste-Test Scale)

• How it works: This method measures exactly how much each ingredient contributed to the final flavor. It calculates the "sensitivity" of the dish to every single word.
• The Result: This was the winner of the study.
• The Analogy: This is like a super-precise scale that tells you, "The salt contributed 10% to the flavor, the sugar contributed 5%, and the pepper contributed 20%." It consistently pointed to the actual "flavor words" (like "amazing" or "terrible") and ignored the boring words. It was stable, reliable, and made sense to humans.

The Big Takeaway

The author ran these tests on a specific type of AI (DistilBERT) looking at movie reviews. Here is what they learned:

• Don't trust the "Eye-Tracker" (Attention): Just because the AI "looks" at a word doesn't mean it's using that word to make a decision. It's often just looking at the grammar.
• Don't rely solely on the "Simulator" (SHAP): While smart, it's too slow and jittery for real-world, fast-paced jobs.
• Use the "Scale" (Integrated Gradients): If you want to know why your AI made a decision, this is the most trustworthy tool. It tells a story that matches human intuition.

The Final Lesson

The paper concludes that Explainability is a diagnostic tool, not a magic truth.

Think of it like a doctor's X-ray. An X-ray helps the doctor see a broken bone, but it doesn't tell the whole story of the patient's health. Similarly, these AI explanation tools help engineers debug their models and build trust, but they aren't perfect. You have to use them wisely, knowing their strengths and weaknesses, rather than blindly believing whatever picture they show you.

In short: If you want to understand your AI, don't just watch its eyes; weigh its ingredients. And for the best results, use the "Integrated Gradients" scale.

Technical Summary

1. Problem Statement

While Large Language Models (LLMs) and transformer-based architectures (e.g., BERT, DistilBERT) achieve state-of-the-art performance in Natural Language Processing (NLP), they operate as "black boxes." Their internal decision-making processes are opaque, creating significant challenges for:

• Trust and Accountability: Deploying models in real-world scenarios where decisions must be justified.
• Debugging: Identifying whether models rely on meaningful patterns or spurious correlations.
• Practical Gap: Although numerous Explainable AI (XAI) methods exist (e.g., attention visualization, gradient attribution, SHAP), there is a lack of empirical understanding regarding how these methods behave in consistent, reproducible, applied settings. Many methods are theoretically sound but may be unstable, computationally expensive, or misaligned with human intuition when applied to transformer-based NLP systems.

2. Methodology

The study employs a controlled, comparative experimental design to evaluate three distinct classes of post-hoc explainability techniques on a specific NLP task.

• Model: A fine-tuned DistilBERT model (a lightweight, efficient variant of BERT).
• Task: Binary sentiment classification using the SST-2 (Stanford Sentiment Treebank) dataset.
• Explainability Techniques Evaluated:
  1. Integrated Gradients (IG): A gradient-based attribution method that measures the contribution of input tokens relative to a baseline.
  2. Attention Rollout: An attention-based method that aggregates attention weights across layers to visualize token interactions.
  3. SHAP (SHapley Additive exPlanations): A model-agnostic method based on cooperative game theory (Shapley values) that treats the model as a black box.
• Evaluation Criteria: The methods were assessed qualitatively and quantitatively based on:
  • Faithfulness: Do the highlighted tokens actually influence the prediction?
  • Stability: Are explanations consistent across repeated runs or similar inputs?
  • Human Interpretability: Do the explanations align with human intuition and domain knowledge?

3. Key Contributions

• Structured Comparative Analysis: Provides a side-by-side evaluation of attention-based, gradient-based, and feature attribution methods within a single, reproducible experimental framework.
• Empirical Validation: Moves beyond theoretical proposals to demonstrate the practical behavior of these tools on a fine-tuned transformer model.
• Practical Guidelines: Offers actionable insights for ML engineers regarding the trade-offs (computational cost vs. reliability) of different XAI tools in production environments.
• Open Resources: The code and trained models are made publicly available to facilitate reproducibility.

4. Key Results and Findings

The study revealed distinct performance characteristics for each method:

• Integrated Gradients (IG):

  • Performance: Emerged as the most reliable technique.
  • Findings: Produced highly stable and intuitive token-level attributions. It consistently highlighted sentiment-bearing words (adjectives, negations, intensifiers) that aligned with human intuition.
  • Verdict: Best suited for debugging and production analysis due to high faithfulness and stability.

• Attention Rollout:

  • Performance: Computationally efficient but less faithful.
  • Findings: While fast to compute, it frequently emphasized structural tokens (e.g., [CLS], punctuation, stopwords) rather than semantically relevant sentiment words.
  • Verdict: Useful for exploratory visualization but unreliable as a standalone explanation for model reasoning.

• SHAP:

  • Performance: Flexible but unstable and costly.
  • Findings: Identified sentiment-relevant components but exhibited high variability across runs due to sensitivity to input formatting and background data. It incurred significantly higher computational overhead.
  • Verdict: Limited scalability in transformer-based text settings; better suited for targeted qualitative analysis rather than routine deployment.

Summary of Trade-offs:

Method	Strengths	Limitations	Best Use Case
Integrated Gradients	High faithfulness, stable, intuitive	Requires gradient access	Production debugging, reliability
Attention Rollout	Fast, easy to compute	Low faithfulness, focuses on structure	Exploratory analysis
SHAP	Model-agnostic, theoretical grounding	High compute cost, unstable	Targeted qualitative analysis

5. Significance and Implications

• Shift in Perspective: The paper argues that explanations should be treated as diagnostic aids rather than definitive accounts of model reasoning. No single method provides a perfect view of the "truth."
• Industry Impact: For ML engineers, the study suggests prioritizing gradient-based methods (like IG) for transformer-based NLP systems where stability and interpretability are critical. It warns against relying solely on attention weights, which are often misleading.
• Future Directions: The authors highlight the need to extend these evaluations to larger, instruction-tuned LLMs, multimodal models, and diverse domains (e.g., medical, legal) to ensure these findings generalize beyond short-text sentiment analysis.

In conclusion, the paper bridges the gap between theoretical XAI research and practical application, demonstrating that while multiple explainability tools exist, Integrated Gradients currently offers the most favorable balance of faithfulness, stability, and usability for transformer-based NLP tasks.