Beyond Topical Similarity: Contrastive Evidence Retrieval with Interpretable Attention Alignment in RAG

Technical Summary: Contrastive Evidence Rationale Attention (CERA)

Problem Statement
Retrieval-Augmented Generation (RAG) systems aim to improve the factuality of Large Language Models (LLMs) by constraining outputs with external documents. However, a critical limitation persists: standard dense retrievers often conflate topical similarity with evidential validity. A retriever may prioritize passages that are lexically related to a query but factually unreliable, contradictory, or merely topically adjacent, rather than those containing specific supporting evidence. This "noisy retrieval" leads to hallucinations and biased outputs, particularly in high-stakes domains like healthcare. Furthermore, existing retrieval systems lack interpretability; they optimize what is retrieved but fail to explain why specific evidence was selected or how it substantiates a decision. Current interpretability methods largely rely on post-hoc explanations rather than intrinsic mechanisms that align model attention with human reasoning.

Methodology
The authors propose Contrastive Evidence Rationale Attention (CERA), a dense retrieval framework designed to inject an "evidential inductive bias" into contrastive learning. CERA operates on two core pillars:

1. Subjectivity-Based Hard Negative Selection:
   Instead of treating all non-relevant documents as equally negative, CERA employs a subjectivity-aware strategy to select "hard negatives." Within a document, chunks that do not overlap with the gold evidence span are ranked by a subjectivity score (computed via TextBlob). The top-$K$ most subjective chunks are selected as hard negatives. This forces the model to distinguish between content that is merely subjective or opinionated versus content that constitutes verifiable, objective evidence, thereby refining the embedding space beyond simple semantic proximity.

2. Interpretable Attention Alignment:
   CERA introduces an auxiliary loss function to supervise the model's internal attention mechanisms.

   • Rationale Supervision: The framework utilizes human-annotated factual rationales (evidence spans) as ground-truth signals.
   • POS-Weighted Masking: A part-of-speech (POS) weighting scheme is applied to these rationales, assigning higher weights to content-carrying tokens (nouns, verbs, adjectives) and lower weights to function words.
   • Alignment Loss: The model minimizes the Kullback-Leibler (KL) divergence between the normalized attention distribution of the [CLS] token to input tokens and the weighted rationale distribution. This forces the attention mechanism to align with the specific tokens that constitute the supporting evidence, rather than relying solely on distributional similarity.

The model is trained using a dual-objective function: a triplet-based contrastive loss ($L_{triplet}$) to separate positive evidence from hard negatives, and the attention alignment loss ($L_{alignment}$), weighted by a hyperparameter $\lambda$.

Key Contributions

1. CERA Framework: The first retrieval framework to combine subjectivity-based hard negative selection with interpretable attention alignment, explicitly injecting an evidential inductive bias into dense retrieval.
2. Rationale-Aligned Attention Objective: A novel training objective that supervises CLS-to-token attention using POS-weighted human rationales. This enables the retriever to identify not just relevant chunks, but the specific tokens constituting evidence, improving faithfulness without sacrificing retrieval performance.
3. Empirical Validation: Comprehensive experiments on the Evidence Inference 2.0 dataset (clinical trial reports) demonstrating that CERA outperforms both the baseline Contriever and standard hard-negative selection methods.

Results
Evaluated on a corpus of 3,393 clinical trial reports, CERA demonstrated significant improvements in retrieval effectiveness and interpretability:

• Retrieval Performance: CERA achieved substantial gains over the hard-negative baseline, with improvements of +0.0768 in Recall@1, +0.1073 in Recall@3, and +0.1307 in Recall@5. The Mean Reciprocal Rank (MRR) increased by +0.1009.
• Interpretability and Faithfulness: The attention alignment objective significantly improved faithfulness metrics. Specifically, the Sufficiency score (measuring how well the explanation reflects the model's decision process) dropped from 0.2073 (Contriever) to 0.0939 under stronger alignment ($\lambda=0.05$), indicating the model relies on more compact and adequate evidence.
• Factuality: Using an ensemble of LLM judges (GPT-5.4, Qwen3-Max, Mistral Large 3) to evaluate the factual relevance of retrieved spans, CERA consistently outperformed the baseline across all ranks, with the Jury Score improving from 1.1881 to 1.9015 at Rank 1.
• Ablation: The study confirmed that subjectivity-based negatives provide a more informative contrastive signal than conventional hard negatives. However, increasing the alignment weight ($\lambda$) beyond 0.05 slightly degraded retrieval metrics, suggesting a trade-off between retrieval precision and attention alignment strength.

Significance and Claims
The paper claims that CERA represents a shift from optimizing purely for "topical similarity" to optimizing for "evidential support." By aligning attention distributions with human-annotated rationales, the framework promotes a more faithful explanation of model behavior, addressing the "black-box" nature of retrieval. The authors assert that this approach is particularly critical for high-stakes domains where unsupported outputs can lead to significant ethical and safety consequences. While the work is a research prototype and not a standalone clinical decision system, it demonstrates that retrieval systems can be trained to identify specific evidence tokens, thereby enhancing both the reliability and interpretability of RAG systems. The authors acknowledge limitations, including the reliance on expert-annotated data and the restriction of experiments to the biomedical domain.