CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHR

Technical Summary of CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHR

Problem Statement
While large language models (LLMs) demonstrate strong performance on medical knowledge benchmarks, their deployment in clinical settings requires capabilities beyond static question answering. Real-world clinical tasks often involve retrospective investigations of heterogeneous, longitudinal electronic health records (EHRs). In these scenarios, agents must determine necessary evidence, retrieve and reconcile structured data with free-text notes, ground conclusions in verifiable evidence, and recognize when a record is insufficient to support a definitive conclusion. Existing benchmarks often fail to test three safety-critical capabilities: negative reasoning (confirming a condition was ruled out), deep longitudinal synthesis across multi-year records, and calibrated abstention (distinguishing between missing evidence and genuine medical ambiguity). Furthermore, few benchmarks evaluate the defensibility of an adjudication process—whether the agent gathered necessary evidence, applied governing standards, and followed required procedures—rather than just the correctness of the final answer.

Methodology
The authors introduce CliniCARE-Bench, a deployment-oriented benchmark for retrospective clinical audit. The framework is built on the following components:

• Data Substrate: The benchmark utilizes MIMIC-IV v3.1, MIMIC-IV-Note v2.2, and MIMIC-IV-ED v2.2, covering inpatient, ICU, and emergency department data from a single academic medical center.
• Scenario Construction: The suite consists of 25 clinician-authored scenarios spanning 14 medical specialties and 10 reasoning capabilities (e.g., time-sensitive protocol auditing, cross-modal validation, longitudinal synthesis). These scenarios are instantiated as 750 patient-specific cases.
• Governed Tool Environment: Agents operate within a reproducible runtime with governed, clinician-verifiable tools. They cannot access raw database schemas directly but must use parameterized tools to retrieve specific record categories (notes, labs, vitals, etc.), perform explicit computations in a sandboxed shell, and access a fixed corpus of governing policies (e.g., KDIGO, CMS SEP-1). Every tool invocation is logged, creating a replayable trace.
• Adjudication Framework:
  • Verdicts: Agents must return one of four verdicts: Yes, No, Indeterminate: Lack of Data, or Indeterminate: Medically Ambiguous. The latter two distinguish between cases where more evidence could resolve the issue versus those requiring expert interpretation even after full review.
  • Reference Labels: Reference verdicts are generated via a multi-model adjudication ensemble (Claude, Codex, Gemini) and calibrated against a blinded review by a Clinical Board.
  • Evaluation Metrics: The framework scores systems on multiple axes beyond raw accuracy:
    • Verdict Correctness: Exact four-class match.
    • Abstention Behavior: Measuring over-commitment (definitive verdicts on indeterminate cases) and over-abstention.
    • Grounding: Citation precision (faithfulness of cited evidence) and evidence coverage (completeness of retrieval).
    • Policy Grounding: Whether agents correctly identify, cite, and apply governing clinical standards.
    • Process Adherence: A weighted rubric of MUST-DO (required actions) and MUST-NOT (prohibited shortcuts) criteria.
    • Defect-Free Accuracy: A metric that credits a verdict only if it is correct and free of prohibited process shortcuts.

Key Contributions

1. Skill- and Specialty-Spanning Scenario Suite: A set of 25 scenarios instantiated as 750 cases, designed to stress-test safety-critical reasoning across diverse medical domains.
2. Governed, Clinician-Verifiable Tool Environment: A reproducible execution surface that isolates query engineering artifacts while preserving record heterogeneity, forcing agents to actively retrieve and reconcile evidence.
3. Grounded, Process-Aware Evaluation Framework: A strict four-way verdict system paired with claim-level evidence grounding, policy citation, and a process rubric that rewards faithful investigation procedures, not just final answer correctness.
4. Scalable Protocol for Clinician-Calibrated Labeling: A method combining clinician-validated specifications with multi-model adjudication and blinded board calibration to produce reliable reference labels at a scale unattainable by exhaustive manual review.

Experimental Results
The authors evaluated 16 agentic systems (10 production agent CLIs on vendor models and 6 open-weight models on a neutral harness). Key findings include:

• Accuracy vs. Process: Raw four-way accuracy ranged from 65.3% to 76.1%. However, defect-free accuracy (correct verdicts without process violations) was 4.8–14.8 percentage points lower. This gap reordered the leaderboard; for instance, a system with high raw accuracy dropped significantly when process defects were penalized.
• Abstention Behavior: All systems systematically under-abstained. The rate of over-commitment (issuing a definitive verdict when the reference required deferral) significantly exceeded the rate of over-abstention. Systems over-committed on 21.3% to 55.3% of deferral cases.
• Grounding vs. Accuracy: There was a dissociation between verdict accuracy and grounding quality. Systems often reached correct verdicts without citing the specific evidence that justified them. Claim-level precision varied by harness family rather than model capability, suggesting the retrieval scaffold is a primary driver of grounding quality.
• Policy Grounding: While systems cited real policy documents with high precision, they frequently omitted governing documents (low recall), citing only a subset of applicable standards.
• Reliability: Repeated attempts showed that systems were often "stably wrong" rather than randomly wrong; high self-consistency did not guarantee correctness, particularly on indeterminate cases.

Significance and Claims
The paper claims that CliniCARE-Bench is the first deployment-oriented clinical agent benchmark to jointly evaluate real longitudinal EHR investigation, claim-level evidence grounding, governing-policy use, process adherence, and calibrated abstention within a common patient-level adjudication framework.

The authors argue that raw accuracy overstates the quality of clinical investigation. A correct final answer can mask defective investigations (e.g., using prohibited shortcuts or missing evidence). The benchmark exposes the "outcome–process gap," demonstrating that for high-stakes clinical audit, the defensibility of the investigation process is as critical as the final verdict. The design principles—governed tool use, explicit adjudication criteria, traceable evidence, and calibrated abstention—are presented as a general recipe for auditable agent environments applicable to other high-stakes domains like finance and law. The benchmark is intended to inform human-supervised deployment by measuring not just if an agent can resolve a case, but how it does so and whether those decisions are transparent and reproducible.