Technical Summary: ClinicalBench and EpiKG

1. Problem Statement

Current clinical reasoning benchmarks (e.g., HealthBench Professional, MedQA) evaluate Large Language Models (LLMs) on "clean" vignettes where facts are pre-supplied. However, real-world Electronic Health Record (EHR) usage requires a preceding step: retrieval faithfulness from messy, longitudinal charts.

In real EHRs, a correct answer depends on navigating:

• Negation: "Patient denies chest pain."
• Temporality: Distinguishing between current conditions, historical resolved conditions, and future hypotheticals.
• Experiencer Attribution: Differentiating between the patient's condition and a family member's history (e.g., "sister had MI").

Existing Retrieval-Augmented Generation (RAG) and Knowledge Graph (KG) systems often flatten this context. They discard assertion metadata (e.g., treating "rule out diabetes" the same as "has diabetes") and fail to route retrieval based on question intent. This creates an epistemic propagation gap, where the loss of assertion state during extraction and storage leads to incorrect reasoning, even if the underlying model is capable.

2. Methodology

2.1 The EpiKG System

The authors propose EpiKG, a patient-level clinical KG-RAG system designed to preserve epistemic metadata end-to-end. It implements three core mechanisms:

1. Epistemic Assertion Schema: A 7-value taxonomy ($\alpha \in \{Pres., Abs., Poss., Cond., Hypo., Fam.Hx., Hist.\}$) extends the i2b2 taxonomy by separating Historical from Family History. This label is propagated through extraction, OMOP mapping, and KG materialization.
2. Bi-Temporal + NLP-Asserted Storage: Edges store valid time, transaction time (Snodgrass tradition), and an NLP-derived temporality label ($\tau_a \in \{Past, Current, Future\}$).
3. Intent-Aware Routing: A classifier maps questions to specific graph traversal strategies:
   • Change: Cross-admission set differencing.
   • Current_State: Filtering for edges with $\tau_a = Current$ or open validity.
   • Historical: Selecting past edges and inferring "resolved" status for conditions present in earlier admissions but absent in the latest.
   • Default: Bidirectional BFS traversal.

2.2 ClinicalBench: The Evaluation Benchmark

To stress-test assertion-aware retrieval, the authors introduce ClinicalBench:

• Cohort: 43 MIMIC-IV patients (convenience sample; 32 with two admissions, 11 with one).
• Scope: 400 questions across 9 assertion-sensitive categories (negation, conditional, uncertainty, family_history, sequence, current_state, duration, historical, change).
• Design: An in-distribution stress test where questions are authored from the same de-identified charts used for retrieval, focusing on retrieval faithfulness rather than cross-site generalization.
• Ablation Conditions:
  • C1: LLM alone.
  • C2/C2b: Vanilla RAG (TF-IDF / Dense/Contriever).
  • C3: KG-RAG without assertions.
  • C4: KG-RAG with assertions but no intent routing.
  • C4g: Intent-Aware KG-RAG (full system).
  • C6: Long Context (all notes concatenated).
  • C7: Deterministic KG (no LLM).

2.3 Evaluation Protocol

• Models: Evaluated across six LLMs (Claude Opus 4.6, GPT-OSS 20B, MedGemma 27B, Gemma 4 31B, MedGemma 1.5 4B, Qwen 3.5 35B).
• Metrics:
  • Primary Endpoint: Leave-author-out paired exact McNemar test on $n=50$ items adjudicated by two external physicians (Hird and Nadeem) with unanimous strict ratings.
  • Sensitivity Endpoints: Deterministic keyword evaluator ($n=362$, change-excluded), three-rater physician majority vote ($n=100$), and oracle intent classification.
• Data Integrity: The authors acknowledge a 56% defect rate in the auto-generated v2 reference answers (due to NLP classifier errors) and report results based on a partially corrected set, with transparency regarding the limitations of the gold standard.

3. Key Results

3.1 Primary Endpoint (Author-Blind)

The leave-author-out paired exact McNemar test on the external physician adjudication ($n=50$) showed:

• C4g vs. C1: $\Delta = +22.0$ percentage points (pp).
• Confidence Interval: 95% Newcombe CI $[+5.1, +31.5]$.
• Significance: $p = 0.0192$.
  This confirms that intent-aware KG-RAG significantly outperforms LLM-alone baselines when evaluated by independent physicians.

3.2 Architectural Novelty and Ablation

• Retrieval vs. Routing: Retrieval alone (C2b, Dense RAG) provided a large gain over C1 (+29.0 pp). However, adding assertions without routing (C4) actually degraded performance compared to C3 (-3.8 pp) on temporal categories.
• The Routing Effect: Intent-aware routing (C4g) recovered and extended the gains.
  • C2b $\to$ C4g (keyword): +8.84 pp ($p = 1.79 \times 10^{-3}$).
  • C2b $\to$ C4g (oracle): +12.43 pp.
• Category Interactions: Assertions alone helped assertion-sensitive categories (e.g., Negation +22.7 pp) but hurt temporal ones (e.g., Historical -30.0 pp). Routing reversed these deficits.

3.3 Cross-Model Convergence

All six models benefited from the C4g architecture. However, the magnitude of improvement was inversely correlated with the baseline strength of the model ($\beta = -1.123, r = -0.921, p = 0.009$). The authors interpret this as regression to the mean rather than evidence that structured retrieval substitutes for model parameter count.

3.4 SliceBench (Complexity Study)

A smaller case study (6 patients, 144 questions) suggested a complexity-dependent effect where KG-RAG gains were larger for patients with more encounters (Tier C: +5.0 pp vs. Tier A: +0.6 pp), though this did not reach aggregate statistical significance.

4. Contributions and Significance

The paper makes three primary contributions:

1. ClinicalBench: A novel, assertion-sensitive benchmark designed to stress-test retrieval faithfulness on real EHRs, exposing category-condition interactions that aggregate scores hide.
2. EpiKG and the Epistemic Propagation Gap: A formalization of the loss of assertion metadata in clinical NLP pipelines and a system implementation that preserves assertion and temporal metadata as first-class properties through retrieval.
3. Architectural Validation: Empirical evidence that intent-aware routing is the critical component for realizing the benefits of structured retrieval. The paper demonstrates that carrying assertion labels without routing can degrade performance, while routing by question intent significantly improves cross-admission clinical QA.

Significance:
The authors argue that while reasoning benchmarks measure the "last mile" of clinical reasoning on clean inputs, the "first mile" is reading the right patient from messy charts. This work posits that representation is the undermeasured prerequisite for clinical AI. The findings suggest that for clinical QA, preserving epistemic metadata (negation, temporality, attribution) and aligning retrieval strategies with question intent are more impactful than simply increasing model size or using long-context windows.

Limitations and Scope:
The paper is modest in its claims:

• It is a single-site, in-distribution stress test, not a multi-site phenotype validation.
• The reference answers have known defects (56% error rate in v2), and the authors explicitly state that the benchmark is a "probe" rather than a definitive measure of clinical deployment readiness.
• The author circularity (designer, builder, and internal evaluator) is mitigated by external physician adjudication and cross-model replication but remains a structural limitation.
• The work does not claim clinical deployment readiness; it identifies governance, safety, and multi-site validation as prerequisites for future deployment.