Technical Summary: Benchmarking Retrieval Strategies for Biomedical RAG

Problem Statement

While Retrieval-Augmented Generation (RAG) offers a mechanism to ground Large Language Model (LLM) outputs in external knowledge, the optimal retrieval strategy for high-stakes domains like biomedicine remains under-explored. In clinical contexts, where accuracy is a patient-safety requirement rather than a design aspiration, the choice of retriever is critical. A retriever returning irrelevant or redundant passages can degrade generation quality regardless of the LLM's capability. Despite the proliferation of retrieval methods—ranging from dense bi-encoders to hybrid and reranking approaches—few studies provide a controlled, multi-metric comparison of these strategies within a single biomedical RAG pipeline with statistically rigorous evaluation.

Methodology

The authors conducted a controlled empirical study comparing five distinct retrieval strategies within a fixed RAG pipeline. To ensure that observed performance differences were attributable solely to the retrieval mechanism, all other components were held constant:

• Dataset: 250 question-answer (QA) pairs randomly sampled from the rag-mini-bioasq subset of the BioASQ benchmark, paired with a corpus of 40,181 PubMed-derived passages.
• Generation Model: GPT-4o-mini (temperature 0.1) with a strict system prompt enforcing grounding in retrieved documents, inline citations, and safety disclaimers.
• Embeddings & Storage: OpenAI's text-embedding-3-small model and ChromaDB vector store.
• Evaluation Framework: DeepEval v1.4.9, utilizing four metrics: Contextual Precision, Contextual Recall, Faithfulness, and Answer Relevancy. Scores were reported as means with 95% bootstrap confidence intervals over the 250 samples.
• Strategies Evaluated:
  1. Dense Vector Search: Baseline k-NN search (k=10).
  2. Hybrid BM25 + Dense: Reciprocal Rank Fusion (weights 0.4/0.6).
  3. Cross-Encoder Reranking: Two-stage pipeline (dense retrieval of k=30, followed by cross-encoder rescoring).
  4. Multi-Query Expansion: LLM-generated query variants (n=3) with deduplicated merging.
  5. Maximal Marginal Relevance (MMR): Diversity-aware selection (λ=0.5) from a fetch pool of k=40.
  6. No-Context Ablation: A lower-bound condition where the generator receives no retrieved context.

Key Results

The study yielded several distinct findings regarding the trade-offs between retrieval strategies:

• Cross-Encoder Reranking Superiority: This strategy achieved the highest composite score (0.827) and the highest Contextual Precision (0.852). The authors attribute this to the cross-encoder's ability to perform token-level attention between query and document, which is crucial for disambiguating biomedical terms (e.g., distinguishing types of "resistance").
• The Dense Baseline: Dense Vector Search performed remarkably well, achieving a composite score of 0.822 (within 0.005 points of the top strategy) and the highest Contextual Recall (0.887).
• The Multi-Query Paradox: Contrary to its design intent to improve recall, Multi-Query Expansion produced the lowest Contextual Precision (0.671) and a lower recall than the dense baseline. The authors hypothesize that for specific biomedical queries, LLM-generated paraphrases introduce retrieval noise rather than resolving ambiguity.
• MMR Trade-offs: MMR sacrificed both Contextual Recall (0.775) and Answer Relevancy (0.658) to achieve diversity, resulting in a lower composite score (0.787).
• Faithfulness Stability: All RAG strategies maintained high Faithfulness scores (0.893–0.902), suggesting that a well-designed grounding prompt effectively suppresses hallucination regardless of the retrieval strategy's quality. The "No-Context" condition showed a high faithfulness score (0.978), which the authors identify as a metric artifact (no context to contradict) rather than a genuine quality signal.
• Value of Retrieval: The "No-Context" ablation collapsed in Answer Relevancy (0.287) compared to all RAG strategies (0.658–0.701), confirming that retrieval is the primary driver of answer quality in this setting, not the model's parametric knowledge.

Key Contributions

1. Reproducible Pipeline: An end-to-end, modular RAG evaluation pipeline where every hyperparameter is version-controlled in a single configuration file, allowing for exact replication.
2. Controlled Empirical Comparison: A head-to-head evaluation of five retrieval strategies on 250 BioASQ QA pairs with per-metric means and 95% confidence intervals, isolating the retrieval variable.
3. Actionable Guidance: Evidence-based recommendations for practitioners, distinguishing between precision-critical and recall-critical use cases in biomedical applications.

Significance and Claims

The paper claims that increased retrieval complexity does not automatically guarantee superior performance at the corpus scale tested. While Cross-Encoder Reranking offers the best overall quality and precision, Dense Vector Search remains a formidable, low-complexity baseline that achieves the highest recall.

The authors emphasize that for precision-critical applications (e.g., clinical decision support), Cross-Encoder Reranking is the recommended choice. However, for recall-critical tasks (e.g., systematic literature review), the simpler Dense Vector Search is optimal. The study also cautions against the default use of Multi-Query Expansion in domain-specific corpora, noting its tendency to degrade precision without improving recall.

Finally, the authors highlight the importance of the "No-Context" ablation as a sanity check, demonstrating that without retrieval, even advanced LLMs fail to provide relevant answers to specific biomedical questions. The work concludes that while retrieval is essential, the choice of strategy must be aligned with the specific constraints of the deployment context (precision vs. recall vs. latency), and that fixed hyperparameters may not generalize across all biomedical query types.