When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering

Technical Summary: When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answering

Problem Statement
Current medical Large Language Models (LLMs) are predominantly trained to encode common, guideline-based medical knowledge within their parameters. Consequently, standard evaluations focus on recalling this memorized content, often through multiple-choice questions. However, real-world clinical practice frequently involves "long-tail" scenarios—rare conditions, atypical presentations, or complex cases that fall outside standard evidence-based guidelines. In these off-guideline situations, relying solely on parametric memory is neither feasible nor reliable. Physicians often must consult external resources, such as published medical case reports, to guide decision-making. Existing benchmarks fail to capture this workflow, as they largely test recall of general knowledge or patient-oriented queries rather than open-ended, expert-level reasoning grounded in specific, rare case evidence.

Methodology
The authors introduce OGCAREBENCH, a benchmark designed to evaluate LLMs on free-form clinical questions requiring reasoning beyond standard guidelines. The construction of this benchmark follows a rigorous four-step pipeline:

1. Data Collection and Filtering: The authors curated a retrieval corpus of 53,617 open-access medical case reports from PubMed Central. To ensure the cases represent persistent rarity and novelty, they filtered out reports published before 2022, those cited more than three times, and those cited by non-case-report articles (indicating the case had been resolved into standard guidelines). This resulted in a subset of 28,219 reports.
2. Extraction of Raw QA Pairs: Using GPT-5.2, the authors extracted three elements from a subset of 1,500 reports: a chronological timeline of the case, the limitations that prevented standard care, and the report's most significant contribution (e.g., a novel diagnostic test or treatment). Questions were formulated to ask for the next appropriate clinical step given the limitations, simulating a scenario where a physician has exhausted standard options.
3. Controlled Modification (Distractors): To prevent models from simply matching the question to the source text via memorization, the authors used Claude 4 Opus to introduce "distractors." These included altering patient demographics, substituting medical terminology with semantically equivalent expressions, and adding comorbidities that do not affect the core condition. The correct answer remained unchanged, ensuring the clinical plausibility of the modified scenario.
4. Expert Verification: Eight physicians in internal medicine or pediatrics validated the modified question-answer pairs. They rated pairs on a 1–5 Likert scale based on medical alignment and the necessity of domain-specific expertise. Only pairs rated 4 or 5 were retained, resulting in a final dataset of 639 instances across 10 medical specialties.

Evaluation Setup
The benchmark evaluates models in two configurations:

• Baseline: Models answer questions using only their parametric knowledge.
• Retrieval-Augmented Generation (RAG): Models are provided with retrieved case reports from the 53,617-report corpus. The authors evaluated 15 retrieval models (sparse, general-purpose, and biomedical) and selected top performers (BM25, BGE-large, and BMRetriever) for the final evaluation. They tested context lengths of 1, 3, and 5 retrieved reports, as well as an "oracle" setting where the ground-truth source report was provided.

Performance was measured using an LLM-as-a-judge (GPT-5.2) to determine if the model's free-text response was "equivalent" to the gold standard answer, a metric validated by human physicians with 93% agreement.

Key Results

• Baseline Performance: Even the best-performing general-purpose model (GPT-5.2) achieved only 56.0% accuracy without retrieval. Medical-specific models performed worse, with MedGemma at 37.1% and Llama 3-Med42 at 42.1%. This indicates that parametric memory alone is insufficient for rare, off-guideline reasoning.
• Retrieval Challenges: The retrieval task itself proved difficult; most retrievers achieved a Recall@1 of less than 50%, meaning the correct source document was missing from the context more than half the time.
• Impact of RAG: Providing retrieved documents significantly improved performance. GPT-5.2's accuracy rose to 82% with 5 retrieved reports, surpassing the oracle performance of specialized medical models. However, a performance gap remained even with oracle documents, indicating that retrieval is necessary but not sufficient; models still struggle to correctly integrate evidence.
• Failure Modes: Analysis of errors revealed that even with the correct document, models frequently failed due to "objective misalignment" (bundling multiple interventions or adding steps beyond the article's scope) and "document grounding" failures (failing to extract the specific action). Specialized models showed weaker extraction of article-specific decisions compared to frontier general-purpose models.

Significance and Contributions
The paper claims three primary contributions:

1. OGCAREBENCH: The introduction of an expert-validated, free-form benchmark derived from published medical case reports, specifically targeting realistic rare clinical scenarios where guidelines are insufficient.
2. Empirical Demonstration of Limitations: The study empirically demonstrates that both general-purpose and medical-specific models struggle with open-ended rare-case reasoning when relying solely on memorization, highlighting the limitations of their standalone use in complex clinical settings.
3. Necessity of Retrieval: The work establishes that retrieval augmentation is essential for building robust medical systems, significantly enhancing performance in expert-level tasks. However, it also notes that retrieval alone does not close the performance gap, emphasizing the need for improved evidence-grounding and reasoning capabilities.

The authors conclude that OGCAREBENCH provides a foundation for benchmarking and advancing LLMs to produce reliable answers in challenging, off-guideline clinical contexts, moving the field beyond simple recall toward evidence-grounded reasoning.