Source-Aware Reranking for Retrieval-Augmented Generation: A Reliability Prior Approach

Technical Summary: Source-Aware Reranking for Retrieval-Augmented Generation

Problem Statement
Standard Retrieval-Augmented Generation (RAG) pipelines typically rely solely on semantic similarity between query and document embeddings to rank retrieved content. This approach treats all documents as equally valid inputs, regardless of their origin, credibility, or potential for harm. In open-domain or uncurated settings, this creates a vulnerability where semantically plausible but factually misleading documents (e.g., AI-generated posts or user forums) can be retrieved with the same priority as peer-reviewed clinical guidelines or government reports. The paper addresses the need for a mechanism to distinguish between high-credibility and low-credibility sources within the retrieval ranking function without altering the underlying language or embedding models.

Methodology
The authors propose a simple, interpretable modification to the standard RAG retrieval pipeline: a source-aware reranking layer that operates post-retrieval but before the final context window. The core mechanism involves assigning a reliability prior, $\lambda(s)$, to each document based on its source type $s$.

The retrieval score is reweighted using the following multiplicative formula:
$$\text{score}_{SA}(q, d) = \text{sim}(q, d) \times \lambda(s_d)$$
Where $\text{sim}(q, d)$ is the standard semantic similarity score and $\lambda(s_d)$ is a predefined domain-informed prior.

Key components of the implementation include:

• Source Taxonomy: Documents are categorized into types such as peer_reviewed, government, established_news, blog, user_generated, and ai_generated.
• Priors: Fixed reliability values are assigned (e.g., 0.95 for peer-reviewed, 0.30 for AI-generated) based on domain assumptions in the health sector.
• Candidate Expansion: To allow the weighting to meaningfully reorder results, the system retrieves an expanded candidate pool (3$k$) before applying the reliability weighting and selecting the top-$k$.
• Optional Extensions: The framework includes optional single-hop neighbor smoothing (adjusting reliability based on nearest neighbors) and a post-query weight update mechanism that adjusts priors based on heuristic feedback (simulated oracle feedback in this study).

Experimental Setup
The framework was evaluated on a controlled corpus of 120 documents in the health and medical domain. The dataset included 10 adversarially crafted documents designed to be semantically close to evaluation queries but containing misleading medical claims. These adversarial documents were assigned low-credibility source types (e.g., ai_generated). The evaluation compared the source-aware system against a similarity-only baseline across 10 medical queries using metrics such as Precision@5, NDCG@5, Mean Reciprocal Rank (MRR), and adversarial hit rate. All experiments were executed on the Rosie high-performance computing cluster at the Milwaukee School of Engineering.

Key Results
In this controlled setting, the source-aware reranking approach yielded the following results:

• Precision Improvement: Precision@5 improved from 0.48 (baseline) to 0.72, a 50% relative increase.
• Adversarial Suppression: The average adversarial hit rate (frequency of low-credibility documents in the top 5) decreased from 0.32 to 0.28.
• Ranking Quality: MRR improved to 1.000, indicating the first retrieved document was relevant for all queries.
• Sensitivity: Sensitivity analysis showed that the magnitude of improvement is driven by the spread between high and low reliability priors. A "compressed" spread yielded modest gains, while the "current" and "aggressive" spreads produced consistent improvements.
• Limitations Observed: The method was not uniformly beneficial; for two specific queries (q006 and q010), the source-aware system performed slightly worse than the baseline, suggesting that topic-specific corpus structures can sometimes cause reweighting to surface adversarial documents that displace other low-reliability clean documents.

Key Contributions
The paper makes three primary contributions:

1. Formulation: A reliability-weighted reranking formulation that functions as a post-retrieval modification, requiring no changes to the underlying language model or embedding model.
2. Evaluation: A controlled adversarial injection evaluation under a specific threat model where low-credibility documents are identifiable via metadata, demonstrating the suppression effect of reliability priors.
3. Empirical Analysis: An analysis on a health-domain corpus suggesting that source provenance weighting can reduce adversarial retrieval and improve precision in controlled environments.

Significance and Claims
The authors explicitly frame this work as an empirical evaluation of a simple modification rather than a proposal of a new retrieval model or a claim of general robustness. The significance lies in demonstrating that lightweight, interpretable source provenance weighting is a viable strategy for mitigating source quality degradation in RAG pipelines within the limits of the experimental setup.

The paper maintains a modest tone regarding its claims:

• It does not claim to solve the general problem of misinformation, noting that an adversary aware of the system could spoof source metadata to bypass the penalty.
• It acknowledges that the reliability priors are manually defined and not learned from data.
• It clarifies that the adaptive learning component relies on oracle feedback (ground-truth labels) and is not a deployable mechanism for real-world user feedback.
• It emphasizes that the results are exploratory due to the small scale (120 documents, 10 queries) and lack of formal statistical significance testing.

Ultimately, the work suggests that incorporating domain-informed source priors is a practical, non-invasive step to improve retrieval reliability, while highlighting the need for future work to address content-level signals, learn priors from data, and test against more sophisticated adversarial threat models.