Technical Summary: Faithfulness Evaluation for Decoder-only LLM Attributions with Controlled Retained Information

Problem Statement

Large Language Models (LLMs), particularly decoder-only architectures like Llama and Mistral, are increasingly evaluated using input attribution methods to explain their decision-making processes. However, comparing the "faithfulness" (reliability) of these explanations remains challenging.

Existing soft-perturbation faithfulness metrics, specifically Soft-NC (Normalized Soft Comprehensiveness) and Soft-NS (Normalized Soft Sufficiency), suffer from a critical bias. These metrics evaluate attribution quality by perturbing inputs based on attribution scores. However, different attribution methods often produce score distributions with different means. Consequently, methods with higher average scores retain more words during soft perturbation, artificially inflating their faithfulness scores simply because the perturbed input contains more information, rather than because the attribution is more accurate. This issue is particularly pronounced in autoregressive decoder-only LLMs, where attribution score distributions can vary significantly across generation steps.

Methodology

1. $\pi$-Soft-NC and $\pi$-Soft-NS

To address the bias in existing metrics, the authors propose $\pi$-Soft-NC and $\pi$-Soft-NS. These are evaluation frameworks that compare attribution methods under a controlled target retaining probability ($\pi$).

• Mechanism: Instead of comparing raw attribution scores, the framework transforms each attribution distribution using an alpha-transformation ($\tilde{s}_i = s_i^\alpha$). The parameter $\alpha$ is numerically solved (e.g., via bisection search) such that the expected retaining probability of the transformed scores matches a specific target $\pi \in [0, 1]$.
• Fair Comparison: By forcing different methods to retain the same expected amount of information (same $\pi$), the evaluation isolates the quality and calibration of the attribution distribution from differences in perturbation strength.
• Evaluation Protocols: The authors introduce two protocols to evaluate different aspects of attribution:
  • Dynamic-Routing: The transformer's attention is dynamically recomputed based on the perturbed input. This measures how attributions identify useful words under changing model conditions.
  • Fixed-Routing: The attention maps are cached from the original unperturbed input and reused during perturbation. This isolates the causal contribution of token content from attention reallocation.
• Visualization: Sweeping $\pi$ from 0 to 1 generates a curve ($\pi$-Soft-NC/NS vs. $\pi$). The Area Under the Curve (AUC) summarizes performance. High scores at low $\pi$ indicate compact sufficient evidence, while strong performance across high $\pi$ indicates broad comprehensiveness.

2. Grad-ELLM

The authors introduce Grad-ELLM, a gradient-based attribution method tailored specifically for autoregressive decoder-only LLMs.

• Inspiration: It adapts the gradient-attention aggregation concept from Grad-ECLIP (designed for dual-encoder vision-language models) to sequential text generation.
• Mechanism:
  • For each decoding step, it computes attribution with respect to the next-token logit.
  • It combines gradient-derived channel importance (sensitivity of the logit to attention output channels) with attention-derived token importance (token weights).
  • To avoid overly sparse "peaky" softmax weights, it employs a "loosened" normalization (min-max normalization) on query-key similarities, resulting in a continuous attribution distribution.
  • Attribution scores are aggregated across selected decoder layers to form the final heatmap.

Key Contributions

1. Identification of Bias: The paper identifies that existing Soft-NC/NS metrics unfairly compare methods with different score means, as they inherently evaluate different amounts of retained information.
2. New Evaluation Metrics: The proposal of $\pi$-Soft-NC and $\pi$-Soft-NS, which normalize comparisons by controlling the expected retained information via a target probability $\pi$.
3. New Attribution Method: The introduction of Grad-ELLM, which integrates gradient and attention signals specifically for the sequential nature of decoder-only LLMs.
4. Comprehensive Empirical Analysis: A rigorous comparison of representative attribution methods (including Attention, Saliency, Integrated Gradients, DeepLIFT, Value Zeroing, and ReAGent) across classification and open-generation tasks.

Experimental Results

Experiments were conducted on Llama-3.1-8B and Mistral-7B (with additional results on 70B) across five datasets (IMDb, SST2, BoolQ, TellMeWhy, WikiBio).

• Diagnosis of Original Metrics: Analysis of raw Soft-NC/NS confirmed the bias; methods like Random and Grad-ELLM retained significantly more tokens (higher $\pi$) than sparse methods like DeepLIFT, leading to incomparable scores.
• Performance under $\pi$-Soft-NC: Grad-ELLM achieved the strongest comprehensiveness-oriented faithfulness (highest average AUC) on both Llama and Mistral. This suggests Grad-ELLM effectively identifies broad evidence coverage.
• Performance under $\pi$-Soft-NS: No single method dominated the sufficiency-oriented evaluation ($\pi$-Soft-NS). Sparse methods like Input$\times$Gradient performed well in the top-3, indicating they may more precisely identify the most critical words, but no method was universally superior.
• Comparison to Random: Most methods only showed an advantage over Random attributions when the retaining probability $\pi > 0.5$. This suggests that for these tasks, significant contextual information (more than just top-ranked tokens) is required, or that current methods struggle with precise ordering of top attributions.
• Qualitative Analysis: In sentiment classification examples, Grad-ELLM successfully highlighted compositional phrases (e.g., "very little positive") rather than isolated tokens (e.g., "positive"), capturing the nuance of negation better than baselines.

Significance and Claims

The paper claims that its proposed evaluation framework provides a rigorous standard for comparing XAI methods for LLMs. By controlling for retained information, $\pi$-Soft-NC/NS allows researchers to distinguish between methods that offer compact sufficient evidence versus those that provide broad comprehensiveness-oriented evidence.

The authors emphasize that Grad-ELLM is designed to produce calibrated, dense attribution distributions rather than extremely sparse rankings. Consequently, while Grad-ELLM excels under the proposed soft-perturbation metrics, it may not be optimal for hard-perturbation ranking metrics (like Insertion/Deletion) where the ordering of a few top tokens dominates. The paper concludes that faithful attribution should be evaluated along multiple axes (sufficiency vs. comprehensiveness) rather than relying on a single metric, and that the proposed metrics support progress in understanding how attribution evidence is organized in autoregressive models.

Limitations acknowledged by the authors:

• Grad-ELLM requires access to internal model states (gradients, attention), limiting it to open-source models.
• The study focuses on faithfulness, not plausibility (human interpretability), which remains difficult to quantify objectively.
• The metrics still rely on embedding-level perturbations, which may not perfectly align with natural text distributions.