💻 computer science

Evaluation Framework for Highlight Explanations of Context Utilisation in Language Models

This paper introduces the first gold standard evaluation framework for assessing highlight explanations of context utilization in language models, revealing that while the adapted mechanistic interpretability method MechLight outperforms existing techniques, all methods struggle with longer contexts and exhibit positional biases.

Original authors: Jingyi Sun, Pepa Atanasova, Sagnik Ray Choudhury, Sekh Mainul Islam, Isabelle Augenstein

Published 2026-04-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jingyi Sun, Pepa Atanasova, Sagnik Ray Choudhury, Sekh Mainul Islam, Isabelle Augenstein

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart robot assistant (a Language Model) that knows a lot of facts from its own memory, like a library it built inside its brain. Sometimes, you give this robot a new piece of paper with information on it and ask it a question. The robot might answer using what it already knows, or it might look at your new paper.

The problem is, the robot doesn't tell you which source it used. Did it read your paper, or did it just guess from its memory?

This paper introduces a new way to test "Highlight Explanations." Think of these explanations as a highlighter pen that the robot uses to mark the exact words in your paper that it read to give its answer. The goal is to see if the highlighter pen is actually pointing to the right words, or if it's just scribbling randomly.

Here is a simple breakdown of what the researchers did and found:

1. The Problem: The "Black Box"

Currently, we can't easily tell if the robot is reading your paper or just relying on its memory. Existing tools that try to show us "what the robot is thinking" (called Highlight Explanations) have never been properly tested to see if they are actually accurate. It's like having a GPS that claims to show you the route, but no one has ever checked if it actually points to the right streets.

2. The Solution: A "Gold Standard" Test

The researchers built a special test lab to check these highlighters. They created four specific scenarios, like a detective game:

The Conflict: The paper says "The capital is Ankara," but the robot's memory says "The capital is London." If the robot answers "Ankara," it must have read the paper. The highlighter should point to "Ankara."
The Distraction: The paper has a lot of nonsense or irrelevant info. The highlighter shouldn't point to the nonsense.
The Double Trouble: You give the robot two different papers that both contradict its memory. The highlighter needs to point to the specific paper the robot chose.

They created a "Gold Standard" (the correct answer key) for these tests so they could objectively grade the highlighters.

3. The Contestants

They tested four different types of highlighters:

The "Blocker" (Feature Ablation): This method tries to hide one word at a time to see if the answer changes. It's like covering a word with your finger to see if the sentence still makes sense.
The "Math Wizard" (Integrated Gradients): This uses complex math to calculate how much each word contributed to the answer.
The "Gaze Tracker" (Attention): This looks at where the robot's internal "eyes" were looking when it spoke.
The "Mechanic" (MechLight): This is a new method the authors created. Instead of guessing, they looked inside the robot's brain (its internal mechanics) to find the specific gear or switch that decided to use the paper, and then traced that back to the words.

4. The Results: Who Won?

The Winner: The "Mechanic" (MechLight) was the best at pointing to the right words. It was the most reliable detective.
The Runners-up: The "Blocker" method was okay but very inconsistent. Sometimes it worked great; other times it got confused.
The Losers: Surprisingly, the "Math Wizard" and the "Gaze Tracker"—which are very popular and used in many other tools—were terrible at this specific job. They often pointed to the wrong words or couldn't tell which paper the robot used.

5. The Big Flaws (The "Gotchas")

Even the best highlighter had two major weaknesses:

The "Long Paper" Problem: As the text got longer, all the highlighters got worse. It's like trying to find a specific needle in a haystack; the bigger the haystack, the harder it is for the highlighter to find the needle.
The "Position Bias": The highlighters had a weird habit of favoring words based on where they were, not what they meant.
- Some methods always liked words at the end of the text (the "recency" bias).
- Others always liked words at the beginning (the "primacy" bias).
- This means if you move a sentence from the start to the end, the highlighter might change its mind about whether that sentence was important, even if the meaning is the same.

6. The Conclusion

The paper concludes that while we have tools to explain how robots use context, most of them are currently unreliable for this specific task. They often fail to show us exactly which part of the text the robot used, especially when the text is long or when there are multiple documents.

The authors' new framework (the test lab) is now available for others to use to build better, more accurate highlighters in the future. They also released their code and data so other scientists can try to fix these problems.

In short: We have a new, rigorous way to test if AI explainers are telling the truth. The test showed that current explainers are often lying (or at least, very confused), but we now have a blueprint to build ones that tell the truth.

1. Problem Statement

Large Language Models (LLMs) are increasingly used in applications requiring the integration of external context (Contextual Knowledge, CK) with internal parametric knowledge (PK), such as Retrieval-Augmented Generation (RAG) and fact-checking. However, a critical transparency gap exists:

Opacity: Users cannot determine if a model's output relies on the provided context or its internal training data.
Lack of Ground Truth: Existing methods for evaluating "Highlight Explanations" (HEs)—which point to specific input tokens responsible for an output—rely on faithfulness metrics (e.g., Comprehensiveness and Sufficiency). These metrics use perturbation proxies (masking/removing tokens) that create out-of-distribution artifacts and lack a gold standard (ground truth) to verify if the explanation actually reflects the model's true source of information.
The Gap: There is no framework to evaluate whether HEs accurately identify which specific context pieces or tokens influenced the model's decision, particularly when the context contradicts the model's parametric memory.

2. Methodology

The authors propose the first gold standard evaluation framework for context attribution, utilizing controlled test cases where ground-truth context usage is predetermined.

A. Controlled Input Regimes

The framework constructs four specific scenarios to test different aspects of context utilization, ensuring the model must use the context (by making the context answer contradict the model's parametric knowledge):

Conflicting (Single): One context passage contradicts the model's PK.
Irrelevant (Single): One context passage is irrelevant but contains a distracting (incorrect) answer token.
Double-Conflicting (Dual): Two context passages both contradict PK.
Mixed (Dual): One irrelevant passage and one conflicting passage.

Note: The authors also create "Swap" versions of dual-context setups to test for positional bias.

B. Evaluation Metrics

The framework assesses HEs across three axes aligned with three Research Questions (RQs):

Document-Level Attribution Accuracy (RQ1 & RQ2):
- $\Delta$ Rank@k: Measures the rank margin of tokens from the "used" document vs. the "unused" document (or context vs. memory). A positive margin indicates the HE correctly prioritizes the relevant source.
- Simulatability: Uses Normalized Mutual Information (NMutInf) and Minimum Description Length (MDL-Bits) to determine if the top- $k$ highlighted tokens can predict the model's source choice (CK vs. PK, or Passage 1 vs. Passage 2).
Token-Level Attribution Accuracy (RQ3):
- Mean Reciprocal Rank (MRR): Evaluates how highly the HE ranks the exact gold answer token(s) found in the context.

C. Evaluated Methods

The framework tests four HE techniques:

Feature Ablation (FA): Masks tokens to measure impact on answer probability.
Integrated Gradients (IG): Uses gradient magnitudes along a path from a baseline to the input.
Attention Visualization (ATTN): Uses attention weights from the most influential attention head.
MechLight (Novel): A mechanistic interpretability approach. It identifies the specific attention head that most strongly drives the logit difference between competing answers (e.g., CK vs. PK) and projects that head's attention weights onto the input tokens as the explanation.

D. Experimental Setup

Datasets: FAKEPEDIA, WORLDCAPITAL, COUNTERFACT, and CONFLICTQA.
Models: Five LLMs ranging from 1.5B to 7B parameters (GPT2-XL, Pythia-2.8B/6.9B, Qwen2.5-3B/7B).

3. Key Contributions

Gold Standard Framework: Introduced the first evaluation benchmark for HEs in context utilization that relies on ground-truth context usage rather than indirect perturbation proxies.
MechLight Method: Proposed a new, lightweight mechanistic interpretability method that converts internal model signals (attention heads driving logit gaps) into token-level highlight explanations.
Comprehensive Benchmarking: Evaluated established methods (FA, IG, ATTN) and the new MechLight method across diverse models, datasets, and context scenarios (single/dual, conflicting/irrelevant).
Public Release: Released the code and the reconstructed dataset to facilitate future research.

4. Results

Performance Comparison

Best Performer: MechLight consistently outperformed all other methods across all scenarios, datasets, and models. It showed the highest accuracy in distinguishing between CK and PK usage and identifying the correct document in dual-context settings.
Second Best: Feature Ablation (FA) showed promise but exhibited high variance and instability, particularly in long-context scenarios.
Poor Performers: Integrated Gradients (IG) and Attention Visualization (ATTN) performed poorly, often yielding margins near zero or negative values. Surprisingly, despite scoring well on traditional faithfulness metrics, they provided little value in revealing actual context usage patterns.

Systematic Failures & Limitations

Context Length Sensitivity: Explanation accuracy degrades significantly as context length increases. All methods struggle to pinpoint the correct answer token in long contexts (e.g., CONFLICTQA).
Positional Bias: In dual-document settings, all methods exhibited strong positional biases:
- FA and IG tended to favor later passages (recency bias).
- ATTN and MechLight tended to favor earlier passages (primacy bias).
- This bias persisted even when the order of documents was swapped, suggesting it is an artifact of the method or model architecture rather than content relevance.
Token Localization Errors: Qualitative case studies revealed that even when models copy answers from context, HEs often highlight:
- Generic tokens (stop words, punctuation).
- Nearby descriptors or entities adjacent to the answer.
- The question phrase itself (e.g., "What", "Who").

5. Significance

Challenges Current Practices: The findings challenge the utility of widely used gradient-based (IG) and attention-based (ATTN) methods for RAG and fact-checking applications, suggesting they may be misleading regarding source attribution.
Foundation for Future Development: The framework provides a rigorous, ground-truth-based foundation for developing the next generation of context attribution methods that must overcome length sensitivity and positional bias.
Mechanistic Interpretability Validation: The success of MechLight highlights the potential of mechanistic interpretability (analyzing internal model components) over standard post-hoc perturbation methods for understanding model behavior in complex retrieval scenarios.

In conclusion, the paper establishes that while current HE methods offer some insight, they are currently insufficient for reliable context attribution in real-world, long-context applications. The proposed framework and the MechLight method offer a path forward for building more transparent and trustworthy LLM systems.