Explainability of Text Processing and Retrieval Methods: A Survey

This paper surveys research on the explainability and interpretability of deep learning-based text processing and information retrieval methods, covering techniques for word embeddings, sequence modeling, attention mechanisms, transformers, BERT, and document ranking while suggesting directions for future work.

The Gist

🕵️‍♂️ The Big Problem: The "Black Box" Mystery

Imagine you go to a magic shop. You ask the wizard, "Show me the best book about cats." The wizard waves a wand, and out pops a perfect book. You are happy, but you ask, "How did you know this was the best one?"

The wizard shrugs and says, "I just... felt it. It's magic."

This is exactly what happens with modern AI search engines and chatbots. For years, computers used simple rules (like "count how many times the word 'cat' appears"). These were easy to understand. But today, we use Deep Learning and Large Language Models (LLMs). These are like super-complex, multi-layered brains with billions of connections. They are incredibly smart and find the best answers, but they are "Black Boxes." We know what goes in (your question) and what comes out (the answer), but we have no idea why they made that specific choice.

This paper is a survey (a big map) of all the research trying to open that black box and say, "Here is why the computer picked that book."

---

🗺️ The Map: What This Paper Covers

The authors looked at hundreds of research papers to organize how we are trying to make these AI systems transparent. They break it down into three main areas:

1. The Old School vs. The New School

• The Old School (Traditional Search): Imagine a librarian who uses a card catalog. If you ask for "cats," they look at the cards, count the words, and hand you a book. You can see the math: "This book has 50 'cat' words, so it's #1." It's transparent.
• The New School (Neural Networks): Imagine a psychic librarian who doesn't use cards. They just "sense" the vibe of the book. They might pick a book with zero "cat" words because it feels right. The problem? We can't see their "sensing" process. This paper focuses on how to explain that "sensing."

2. The Two Types of Explanations

The authors categorize the methods used to explain AI into two buckets:

• The "Surrogate" Method (The Shadow Puppet):
  Imagine you have a complex, scary robot (the AI). You can't understand it. So, you build a simple, friendly puppet (a simple model) that mimics the robot's movements.

  • Analogy: If the robot raises its left hand, the puppet raises its left hand. By watching the simple puppet, you can guess what the robot is doing.
  • In the paper: Researchers build simple, easy-to-understand models to mimic complex AI rankings. If the simple model agrees with the complex one, we trust the explanation.

• The "Feature Attribution" Method (The Highlighter):
  Imagine you are reading a long essay. You want to know why the teacher gave it an 'A'. You use a highlighter to mark the specific words that made the grade.

  • Analogy: "The AI liked this document because it highlighted the words 'sustainable' and 'energy'."
  • In the paper: This is called LIME or SHAP. It asks, "If I remove the word 'energy' from this document, does the AI still like it?" If the score drops, that word was important.

3. The New Frontier: RAG (Retrieval-Augmented Generation)

This is the hottest topic right now. Imagine an AI chatbot that doesn't just "know" things from its training, but has a library it can look up in real-time.

• The Problem: The AI reads a book from the library, writes an answer, and cites the book. But did it actually read the book, or did it just guess based on what it already knew?
• The Paper's Focus: How do we know the AI is being faithful? Is it really using the library, or is it hallucinating? The paper reviews tools that check if the AI's answer is actually grounded in the text it retrieved.

---

🧪 The Toolkit: How Do We Test These Explanations?

The authors point out a major issue: How do we know an explanation is good?

• The "Human" Test: Ask a person, "Does this make sense?"
  • Problem: Humans are biased. Sometimes we think an explanation is good just because it sounds nice, even if the AI didn't actually use that logic.
• The "Fidelity" Test: Does the explanation accurately mimic the AI's brain?
  • Analogy: If the AI says "Blue," and the explanation says "Red," the explanation is bad.
• The "Sufficiency" Test: If we only give the AI the "highlighted" words, can it still give the right answer?
  • Analogy: If you take away the "cat" words from the document, does the AI stop recommending it? If yes, the explanation is solid.

The Big Takeaway: The paper admits that right now, we don't have a perfect "ruler" to measure these explanations. It's a bit like trying to measure the taste of a soup without a tongue. We are still figuring out the best way to test if an AI is telling the truth about why it made a decision.

---

🚀 What's Next? (Future Directions)

The authors end with a list of things we still need to figure out:

1. The "Lost in the Middle" Problem: Imagine you give an AI a 10-page document to read. It seems to only read the first page and the last page, ignoring the middle. We need to explain why it ignores the middle and how to fix it.
2. The "Right Answer, Wrong Reason" Trap: Sometimes an AI gives the correct answer but cites the wrong document. It's like a student getting the math right but showing the wrong work. We need to catch this.
3. Standardized Tests: We need a universal "driver's license" test for AI explanations. Right now, every researcher uses a different test, making it hard to compare who is doing the best job.

🎯 Summary in One Sentence

This paper is a guidebook for researchers trying to turn the mysterious, "black box" AI search engines into transparent, understandable tools, ensuring that when the computer gives us an answer, we know exactly why it chose that answer and that it's not just guessing.

Technical Summary

Here is a detailed technical summary of the survey paper "Explainability of Text Processing and Retrieval Methods: A Survey" by Sourav Saha, Debapriyo Majumdar, and Mandar Mitra.

1. Problem Statement

The rapid adoption of Deep Learning (DL) and Large Language Models (LLMs) in Information Retrieval (IR) and Natural Language Processing (NLP) has revolutionized text processing, particularly in document ranking and Retrieval-Augmented Generation (RAG). However, these models rely on complex, non-linear architectures (e.g., Transformers, dense vector spaces) that function as "black boxes."

• The Core Issue: Unlike traditional IR models (e.g., BM25) which use human-interpretable features (term frequency, document length), modern neural models convert text into dense vectors in high-dimensional spaces ($\mathbb{R}^n$) where dimensions lack semantic meaning.
• The Need: There is a critical need for Explainable IR (ExIR) and Explainable NLP (ExNLP) to provide human-understandable justifications for model decisions. This is essential for debugging, building user trust, ensuring fairness, and meeting regulatory compliance.
• Gap: While general surveys on XAI exist, there was a lack of comprehensive, up-to-date surveys specifically focusing on the intersection of IR, NLP, and the emerging RAG paradigm, particularly covering recent advancements post-2022.

2. Methodology and Scope

The authors conducted a systematic survey of literature from 2004 to 2025, covering top-tier venues in IR (SIGIR, CIKM, WSDM), NLP (ACL, EMNLP), and ML (NeurIPS, ICML).

• Classification Framework: The paper organizes explainability research using a multi-dimensional schema:
  1. Model-Agnostic vs. Model-Aware: Whether the explanation relies on the internal structure of the model.
  2. Inherent vs. Post-hoc: Whether the model is designed to be interpretable or explained after training.
  3. Scope: Global (overall model behavior) vs. Local (specific input-instance decision).
  4. Form: Surrogate models, feature attribution (heatmaps, rationales), and contrastive explanations.
• Structure: The survey is divided into core sections covering Document Ranking, RAG systems, and supplementary appendices detailing explainability for specific components (Embeddings, Sequence Models, Attention, Transformers/BERT).

3. Key Contributions and Technical Findings

A. Document Ranking (Sections 6.1 – 6.9)

The survey categorizes explanations for Neural Ranking Models (NRMs) into global and local approaches:

• Global Explanations:
  • Axiomatic Framework: Uses formal constraints (axioms) like Term Frequency (TFC) and Document Length (LNC) to diagnose why models rank documents as they do. Studies show that while NRMs often outperform traditional models in effectiveness, they frequently violate these axioms (e.g., failing to penalize long documents correctly), suggesting a disconnect between neural learning and established IR heuristics.
  • Probing: Techniques like "probing classifiers" are used to determine what linguistic features (POS, NER, syntax) are encoded in intermediate layers of BERT-based rankers. Findings suggest that while early layers capture syntax, later layers focus on semantic relevance, though this varies by dataset.
  • Additive Models: Approaches like Generalized Additive Models (GAMs) attempt to approximate complex NRMs with interpretable linear or tree-based models to understand feature contributions.
• Local Explanations:
  • Feature Attribution: Adaptations of LIME and SHAP (e.g., LIRME, EXS, RankSHAP) are used to assign importance scores to specific tokens or terms.
  • Rationales: Methods that extract specific spans (sentences or phrases) from a document that justify the relevance score. The survey highlights that while rationales improve interpretability, they often suffer from low faithfulness (the rationale doesn't actually drive the model's decision).
  • Contrastive/Counterfactual: Techniques that modify queries or documents minimally to change the ranking order, identifying the "critical" features responsible for the decision.

B. Retrieval-Augmented Generation (RAG) (Section 7)

A significant portion of the survey addresses the explainability of RAG systems, which combine LLMs with external retrieval.

• Faithfulness & Attribution: The core challenge is determining if an LLM's answer is grounded in the retrieved context or its internal parametric memory.
  • Metrics: Introduction of metrics like Knowledge-F1 (lexical overlap) and model-based frameworks like RAGAs and ARES (using LLM judges) to evaluate faithfulness.
  • Attribution: Methods to trace specific tokens in the answer back to specific tokens in the retrieved documents (e.g., MIRAGE, self-citation).
• Knowledge Conflicts: The survey discusses "Context-Memory Conflicts" where retrieved evidence contradicts the LLM's training data. New benchmarks and adaptive decoding mechanisms are proposed to resolve these conflicts.

C. Component-Level Explainability (Appendices B–E)

The paper provides a deep dive into the interpretability of specific architectural components:

• Embeddings: Techniques to transform dense vectors into interpretable spaces (e.g., POLAR, Densifier) or modify objective functions to align dimensions with semantic concepts.
• Sequence Models: Analysis of LSTMs using contextual decomposition to identify how specific phrases influence predictions.
• Attention Mechanisms: A critical debate is presented: Is Attention Explanation? The survey cites evidence (Jain & Wallace, Serrano & Smith) suggesting attention weights are often not faithful explanations of model behavior, as permuting them rarely changes the output. However, methods to enforce "faithful attention" via loss function modifications are discussed.
• Transformers/BERT: Analysis of layer-wise information flow, showing that lower layers capture syntax while higher layers capture semantics. The survey notes that "skip connections" play a more crucial role in information flow than attention weights alone.

4. Results and Evaluation Challenges

• Evaluation Metrics: The paper highlights a lack of standardized evaluation for ExIR. Current metrics include:
  • Fidelity: How well a surrogate model mimics the black box (e.g., Rank Correlation).
  • Sufficiency/Comprehensibility: Whether the explanation alone is enough to reproduce the prediction.
  • Human Evaluation: Often anecdotal due to resource constraints, though user studies are increasing.
• Key Findings:
  • Traditional IR models are inherently more explainable but less effective than NRMs.
  • NRMs often violate axiomatic principles of IR, suggesting they learn different (and sometimes brittle) patterns.
  • Attention weights are poor proxies for feature importance; gradient-based methods (Integrated Gradients) and perturbation-based methods (LIME/SHAP) are generally more reliable.
  • In RAG, models frequently exhibit "right for the wrong reasons" behavior, citing relevant documents even when the answer was generated from internal memory.

5. Significance and Future Directions

• Significance: This survey serves as a definitive reference for researchers entering the field of ExIR, bridging the gap between traditional IR theory and modern deep learning. It consolidates scattered research on RAG explainability, a rapidly evolving sub-field.
• Future Research Directions:
  1. Standardized Evaluation: The authors call for a unified evaluation framework (protocols and test collections) similar to TREC for standard IR, to allow reproducible comparison of explanation methods.
  2. Domain-Specific Explainability: More work is needed in high-stakes domains like legal and medical retrieval, where "black box" decisions are unacceptable.
  3. RAG Dynamics: Investigating how variations in the retrieval depth and ranking order impact the factuality and explainability of the final generated response.
  4. Negative Sample Analysis: Understanding how negative training samples influence the loss function and model behavior in ranking tasks.

In conclusion, the paper argues that while Deep Learning has improved retrieval effectiveness, the field must urgently prioritize faithful, human-centric explanations to ensure these powerful systems are transparent, reliable, and trustworthy.