Technical Summary: Towards Error-Free EHRs

Problem Statement

Electronic Health Records (EHRs) rely on the interdependence of two primary data modalities: structured tables (e.g., vital signs, prescriptions, lab results) and unstructured clinical notes (e.g., physician notes, discharge summaries). While clinicians synthesize these sources for decision-making, discrepancies frequently arise due to administrative system architectures and documentation practices. These inconsistencies pose significant risks to patient safety and create legal liabilities.

Existing automated approaches for verifying note-table consistency, such as the prior work EHRCon, primarily rely on surface-level matching. These methods check if specific numeric values or discrete events mentioned in notes appear in structured tables. However, they fail to capture the reasoning-intensive nature of real-world clinical documentation, which often involves:

1. Clinical Interpretation: Notes describe interpreted states (e.g., "anemia") that require verifying if underlying measurements (e.g., hemoglobin levels) meet specific clinical criteria.
2. Event Relations: Notes describe relationships among multiple events (e.g., a drug administered because of a specific lab trend), requiring validation of event relationships rather than isolated entries.
3. Temporal Dynamics: Notes describe changes over time (e.g., "BP declined steadily"), requiring the assessment of trends and time spans rather than single data points.

The lack of benchmarks and frameworks capable of handling these reasoning-intensive aspects limits the reliability of automated consistency verification.

Methodology

1. EHR-ReasonCon: A Reasoning-Intensive Benchmark

To address the data gap, the authors introduce EHR-ReasonCon, a high-quality benchmark built upon the MIMIC-III dataset.

• Scale and Composition: The dataset comprises 8,048 annotated entities derived from 105 clinical notes (discharge summaries, physician notes, and nursing notes) linked to 14 structured tables.
• Annotation Protocol: The annotation process was guided by four clinical experts and involved a rigorous multi-stage protocol:
  • Tool Development: Eight specialized table-exploration tools were created to support systematic evidence retrieval. These tools fall into three categories: Entity-to-Table Item Alignment (handling lexical/semantic variations), Database Exploration and Value Profiling (understanding schema and value distributions), and Temporal and Conditional Record Retrieval (querying trends and time windows).
  • Process: Annotators identified anchor entities, used tools to retrieve structured evidence, and performed reasoning-intensive verification involving temporal reasoning, commonsense reasoning, and medical interpretation.
  • Quality Control: A dual-annotation process with physician adjudication ensured high reliability, achieving inter-annotator agreement scores of 0.897 for Named Entity Recognition (NER) and 0.888 for consistency labeling.
• Ground Truth: The dataset provides high-quality labels distinguishing between Consistent and Inconsistent entities, with a focus on reasoning-intensive discrepancies (e.g., time errors, value errors, and missing evidence).

2. EHR-INSPECTOR: An LLM-Based Framework

The authors propose EHR-INSPECTOR, an LLM-based framework designed to mirror the human annotation workflow for consistency verification. It operates through four sequential modules:

1. Note Segmentation and Temporal Reference Extraction: The framework segments clinical notes into topic-coherent spans and resolves relative temporal expressions (e.g., "Day 3 post-op") into absolute timestamps using global temporal references.
2. Multi-Stage LLM-Based Entity Extraction: To avoid computational inefficiency from searching the entire EHR ontology (~14,000 items), the framework employs a two-step strategy:
   • Patient-Specific Extraction: Extracts entities aligned with items already recorded for the specific patient.
   • Ontology-Guided Hierarchical Extraction: Uses a pre-constructed hierarchical ontology to narrow the search space for entities not in the patient's specific records, extracting relevant high-level groups and subgroups before identifying specific items.
3. Temporal Reasoning and Scope Filtering: Entities are filtered to retain only those associated with the current hospitalization (Active), excluding past history and future plans to prevent false inconsistency flags.
4. Tool-Augmented Consistency Verification: The LLM interacts with structured tables via the eight table-exploration tools defined in the benchmark. To improve efficiency, a Validation Cache stores recently verified entities and their evidence, allowing related entities (e.g., "soft" and "distended" from a single abdominal exam) to share verification results and avoid redundant tool calls.

3. Evaluation Strategy

The framework is evaluated using an LLM-as-a-judge paradigm, validated against clinical expert judgments. Two evaluation criteria are used:

• Harsh: Enforces exact agreement with gold annotations.
• Lenient: Accepts clinically plausible variations where the reasoning is sound, even if the specific entity span or table mapping differs from the gold standard.

Key Results

Performance

Evaluated across multiple model backbones (including Gemini 2.5 Flash, Qwen3 32B, MedGemma 27B, and GPT-OSS 20B), EHR-INSPECTOR consistently achieves state-of-the-art performance compared to the baseline CheckEHR.

• Improvement: The framework substantially improves both recall and precision. For instance, using Gemini 2.5 Flash, EHR-INSPECTOR achieved a Lenient F1 score of 74.75 compared to CheckEHR's 67.27.
• Reasoning Capability: The performance gap widens with models possessing stronger reasoning capabilities, suggesting that the framework effectively leverages advanced reasoning to handle complex consistency checks.
• Generalization: EHR-INSPECTOR demonstrates strong generalization, outperforming baselines on the surface-level EHRCon dataset and maintaining robust performance on MIMIC-IV and perturbed schema variations.

Ablation and Analysis

• Component Contributions: Ablation studies confirm that note segmentation, scope filtering, and the specific entity extraction strategy are critical for performance. Removing segmentation or filtering leads to significant F1 score drops.
• Tool Usage: Analysis of tool usage traces reveals that while human annotators follow optimized, conservative paths, LLMs tend to explore broader, more dispersed paths. Interestingly, removing certain "space-reduction" tools (designed for human working memory constraints) sometimes improved LLM performance, suggesting agents should dynamically manipulate toolsets.
• Error Analysis: The most frequent error across models is "Premature Conclusion," where models terminate verification before gathering sufficient evidence. Stronger models exhibit fewer validation cache errors, indicating better tracking of intermediate states.

Significance and Claims

The paper claims that EHR-ReasonCon and EHR-INSPECTOR represent a significant step forward in automating EHR consistency verification by moving beyond surface-level matching to reasoning-intensive verification.

• Bridging the Gap: The work addresses the critical gap between the nuanced, interpretive nature of clinical notes and the rigid structure of EHR tables, which existing methods fail to capture.
• Reliability: By incorporating expert-guided annotations and specialized tool-augmented reasoning, the framework provides a more reliable mechanism for detecting discrepancies that could compromise patient safety.
• Scalability: The use of LLMs with tool-augmented workflows offers a scalable alternative to impractical manual verification, capable of handling complex temporal and relational clinical data.

The authors acknowledge limitations, including the reliance on a specific dataset (MIMIC-III) which may not fully represent all real-world clinical scenarios, and the inherent risks of model-driven biases in LLM-as-a-judge evaluation. They suggest future work should focus on expanding data diversity and incorporating expert-in-the-loop frameworks to further enhance robustness.