Technical Summary: MamaBench and Evidence-Anchored RAG

1. Problem Statement

Current medical Large Language Model (LLM) benchmarks (e.g., MedQA, PubMedQA) evaluate questions in isolation. This approach fails to detect a critical failure mode in clinical AI: Diagnostic Fixation. A model may correctly diagnose a base case but fail to adjust its diagnosis when a single discriminative clinical parameter changes, leading to the same incorrect intervention for a clinically distinct presentation. This fragility is particularly dangerous in maternal and child health, where diagnostic errors threaten two lives. Existing counterfactual benchmarks (e.g., MedEinst) do not specifically address this domain, and standard Retrieval-Augmented Generation (RAG) systems suffer from "retrieval stagnation," where base and counterfactual queries yield near-identical embeddings and thus identical retrieved contexts, failing to surface the specific evidence needed to distinguish the cases.

2. Methodology

2.1 MamaBench Benchmark

The authors introduce MamaBench, the first counterfactual benchmark for maternal and paediatric AI.

• Construction: The dataset comprises 217 counterfactual pairs (434 total cases) covering 371 unique pathologies. Unlike benchmarks derived from existing datasets or LLM pipelines, every narrative is expert-authored by a three-member clinical team (obstetrics, gynaecology, paediatrics) in a first-person, patient-reported format to introduce naturalistic ambiguity.
• Counterfactual Generation: For each base case, a counterfactual is created by minimally perturbing parameters across five categories: symptom substitution, severity escalation, risk factor modification, temporal shift, and comorbidity introduction.
• Evaluation Metrics:
  • Base Accuracy ($Acc_{base}$): Standard accuracy on the base cases.
  • Bias Trap Rate (BTR): The primary metric, defined as the conditional probability that a model fails the counterfactual case given it succeeded on the base case ($BTR = \frac{|PF|}{|PP| + |PF|}$). This quantifies Diagnostic Fixation.
  • Robust Accuracy ($Acc_{rob}$): The accuracy on both base and counterfactual cases.
  • Judging: Predictions are evaluated by an LLM-as-judge (Claude Opus 4.5), which is lenient on nomenclature but strict on clinically significant modifiers affecting management.

2.2 Evidence-Anchored RAG (EA-RAG)

To address retrieval stagnation, the authors propose EA-RAG, a three-stage inference-time pipeline that replaces aggregate similarity retrieval with an evidence coverage objective.

• Stage 1: Evidence Extraction: A structured LLM call parses the patient narrative into typed clinical parameters (demographics, severity, risk factors, etc.). Null fields are treated as informative signals.
• Stage 2: Two-Pass Retrieval with Coverage Auditing:
  • Pass 1: Retrieves top-$k$ chunks via standard cosine similarity.
  • Audit: Scores each extracted evidence element against the retrieved set. Elements with low coverage ($< \theta$) form a "gap set."
  • Pass 2: Generates contrastive sub-queries for the gap set (e.g., "How does [parameter] change management?") to retrieve supplementary chunks. The final context is capped at $k+3$ chunks.
• Stage 3: Taxonomy-Grounded Generation: The generator receives the narrative, the enriched context, and a scaffold instruction set derived from an analysis of baseline failure modes (e.g., "Missed critical info," "Underthinking"). This scaffold enforces a clinical-reasoning-first hierarchy, requiring the model to diagnose from the presentation before consulting evidence and permitting the override of retrieved context if clinical reasoning diverges.

3. Key Contributions

1. MamaBench: A novel, expert-authored counterfactual benchmark for maternal and paediatric care, formalizing the concept of Diagnostic Fixation and introducing the Bias Trap Rate (BTR) as a metric for robustness.
2. EA-RAG: A retrieval intervention that operates entirely at inference time (no fine-tuning) to ensure discriminative parameters are surfaced in the context, even when query embeddings are similar.
3. Empirical Analysis: A comprehensive evaluation across eight configurations of four frontier LLMs (GLM-5, Kimi K2.5, GPT-4o/5.4, Claude Sonnet 4.6) demonstrating the limitations of current clinical AI.

4. Results

• Systemic Fragility: Across all models, base accuracy significantly overstates robust accuracy. The gap ranges from 16 to 28 percentage points. For instance, GPT-4o achieved 77.4% base accuracy but only 46.5% robust accuracy (BTR = 39.9%).
• Limitations of Standard RAG: Vanilla RAG provides no counterfactual benefit; its BTR matches standalone models of similar capability, confirming that retrieval stagnation is the bottleneck.
• EA-RAG Performance:
  • On the strongest model (Claude Sonnet 4.6), EA-RAG reduced BTR from 25.8% to 20.3% (a 5.5 percentage point reduction) while maintaining base accuracy (81.6% vs. 82.0%).
  • Robust accuracy improved from 60.8% to 65.0%.
  • Ablation studies confirmed that while coverage auditing alone was insufficient, the combination of coverage-aware retrieval and the taxonomy-grounded scaffold yielded the best results, with the scaffold driving the largest single gain in reasoning quality.
• Residual Challenge: Even the best system (Claude + EA-RAG) fails on approximately 20% of counterfactual pairs, indicating that counterfactual robustness remains an open challenge.

5. Significance and Claims

The paper claims that base accuracy is a misleading metric for clinical AI deployment, as it systematically overstates competence by hiding diagnostic fixation. The introduction of MamaBench provides the first tool to measure this specific failure mode in maternal and child health.

The authors assert that retrieval quality is the primary bottleneck for robustness in clinical RAG systems. While EA-RAG demonstrates that evidence coverage objectives can significantly mitigate this issue without degrading base performance, the residual 20% BTR confirms that retrieval-level interventions alone are insufficient to solve the problem. The paper concludes that achieving true counterfactual robustness in clinical AI requires further research beyond current retrieval and prompting strategies.

Limitations acknowledged by the authors:

• The benchmark is limited to a single clinical domain (maternal/paediatric) and English language.
• Evaluation relies on an LLM-as-judge, which may introduce systematic biases correlated with the judge model's own blind spots.
• The reliance on expert authoring limits the speed of scaling the benchmark to new domains.