Re-Ranking Through an Attribution Lens for Citation Quality in Legal QA

Technical Summary: Re-Ranking Through an Attribution Lens for Citation Quality in Legal QA

Problem Statement
Retrieval-augmented generation (RAG) systems for legal question answering (QA) typically rely on semantic similarity to retrieve passages, assuming that top-ranked documents are the most useful for the language model (LM) to cite. However, this assumption fails in specialized domains like European Court of Human Rights (ECHR) case law. On the AQuAECHR benchmark, semantic similarity between a query and retrieved passages shows effectively no correlation with perturbation-based attribution scores, which measure the actual causal influence of a passage on the generated answer. Consequently, standard retrievers often surface topically related but doctrinally irrelevant precedents, causing the generator to ignore high-ranked evidence or "post-rationalize" claims using parametric memory rather than the provided context. Furthermore, semantic similarity-based ranking performs worse than random selection at surfacing "gold" citation paragraphs within the candidate pool.

Methodology
The authors propose a pipeline that repurposes perturbation-based attribution methods as a training signal for passage re-ranking, rather than using them solely for post-hoc explanation. The approach consists of three stages:

1. Attribution Scoring: For each question, the top 10 passages are retrieved using a GTR-T5-XL dense bi-encoder. A language model (Mistral-7B or Llama-3-8B) generates an answer, and C-LIME (a linear perturbation-based attribution method) is applied to compute continuous scores for all passages. These scores quantify how much the generated output changes when a specific passage is removed, distinguishing passages the model relies on from those it ignores.
2. Cross-Encoder Training: A lightweight cross-encoder (ms-marco-MiniLM-L-6-v2) is trained to predict these continuous C-LIME scores from question-passage pairs. The model is trained using mean squared error loss against the attribution targets. This process is repeated for each language model and via five-fold cross-validation, resulting in ten distinct re-rankers.
3. Re-Ranking and Generation: During inference, the retriever fetches the top 50 passages. The trained cross-encoder re-scores these candidates, and the top 10 are passed to the language model for answer generation. This step adds minimal latency (<1% of total time) and requires no additional LLM calls.

Key Contributions

1. Attribution as a Training Signal: The work transitions perturbation-based attribution from a post-hoc explanation tool to a training signal for passage re-ranking. This results in improved citation faithfulness and alignment with gold expert answers.
2. Architecture vs. Training Signal: The authors demonstrate that the performance gains stem specifically from training on attribution scores, not merely from the cross-encoder architecture. A pretrained cross-encoder without C-LIME training captures less than half of the citation gains achieved by the attribution-trained model.
3. Cross-Model Convergence: Two re-rankers trained independently on different language models (Mistral and Llama) converge to a high agreement (Spearman correlation of 0.858) despite their raw attribution scores agreeing at only 0.480. This indicates the cross-encoder reduces model-specific noise and learns a shared relevance signal that transfers across models, though same-model re-ranking remains more effective.

Results
Evaluated on the AQuAECHR benchmark (1,116 questions), the C-LIME-trained re-ranker consistently outperforms the baseline and a pretrained-only re-ranker across all six metrics:

• Citation Faithfulness: Improved by +5.02 percentage points (pp) for Mistral and +2.70pp for Llama.
• NLI-based Citation Similarity: Improved by +6.67pp (Mistral) and +5.72pp (Llama).
• Gold Citation Retrieval: Under standard similarity ranking, gold paragraphs had a mean rank of 37.4 out of 50. Re-ranking shifted this to 11.1 (Mistral) and 12.0 (Llama).
• Answer Correctness: Metrics such as ROUGE-L, BERTScore, and Claim Recall saw negligible changes (<0.5pp), indicating the re-ranker improves citation quality without altering the core answer content.
• Cross-Model Transfer: Re-rankers trained on one model and applied to the other retained 43–51% of the same-model citation gains, confirming the existence of a transferable relevance signal.

Significance and Claims
The paper claims that the gap between retriever relevance and generator citability can be narrowed by training on behavioral measures of passage influence (attribution) rather than surface relevance or self-reported rationales. The authors emphasize that a single offline attribution run can generate training data for a re-ranker that improves citation quality at inference time with no additional cost.

However, the authors remain modest regarding the broader implications. They note that their evaluation relies on automatic NLI-based metrics and that legal expert assessment is required to validate if these improvements translate to meaningful gains for practitioners. They acknowledge a risk that re-ranking based on generator behavior might reinforce existing model biases rather than promoting objectively relevant passages, though the cross-model convergence finding suggests a shared signal exists. The work is framed as a proof of concept for the "attribution-to-re-ranking" paradigm within the ECHR domain, with future generalizability to larger models and other jurisdictions left as an open question.