A Generative Approach for Semantic Auditing of Electronic Health Records

Technical Summary: A Generative Approach for Semantic Auditing of Electronic Health Records

Problem Statement
The reliability of clinical artificial intelligence (AI) is fundamentally dependent on high-quality data. However, Electronic Health Records (EHRs) frequently exhibit inconsistencies with existing scientific knowledge. Current quality assessment methods are limited in scope: they predominantly focus on syntactic completeness (e.g., ensuring fields are not null or formatted correctly) or rely on labor-intensive, manually curated rules to capture semantic nuances. These manual approaches do not scale to the complexity of medical data and often fail to detect "Semantic Gaps"—instances where data is syntactically valid but epidemiologically implausible. Furthermore, existing tools often neglect contextual factors such as specific populations, geographies, and time frames, leading to the "Garbage in - Garbage out" phenomenon where poor data quality undermines downstream research and clinical model validity.

Methodology: Medical Data Pecking
To address these scalability and semantic limitations, the authors propose Medical Data Pecking, a methodology that adapts software engineering unit testing principles for medical data validation. The core innovation is the introduction of Semantic Data Coverage, which utilizes Large Language Models (LLMs) to generate context-aware tests that "peck" for inconsistencies between observed data and epidemiological evidence.

The framework, implemented as the Medical Data Pecking Tool (MDPT), operates through a decoupled architecture to ensure data privacy and security:

1. Generation Module: Operates on study specifications and data dictionaries (metadata) without accessing raw patient data. It uses an LLM to map clinical intents to local database schemas.
2. Retrieval-Augmented Generation (RAG): The system employs a two-stage process for test synthesis:
   • Contextual Parameter Extraction: The system identifies relevant medical concepts and aligns them with standard vocabularies (e.g., OMOP, ICD-9, SNOMED) using vector search.
   • Automated Grounding: It retrieves epidemiological statistics via web search (Bing API), prioritizing high-authority sources (e.g., CDC, NIH).
3. Auditor Agent (Double-Pass Verification): To mitigate LLM hallucinations, a secondary "Auditor Agent" performs an independent, targeted search to verify the statistics proposed in the initial test matrix. This agent corrects discrepancies or discards unsupported claims before final test generation.
4. Test Application Module: Resides within the user's secure environment, executing the generated test suite against raw patient data.

The framework defines a hierarchical taxonomy of Semantic Unit Tests:

• Syntactic Level (Metadata): Verifies structural integrity (field presence, data types).
• Distributional Semantic Level: Validates that population metrics align with theoretical or literature baselines (e.g., expected prevalence ranges).
• Contextual Semantic Level (Subpopulation): Analyzes intra-group patterns (e.g., comorbidity rates within specific demographics) to detect context-specific biases invisible to aggregate analysis.

Statistical validation employs three criteria: Welch's t-test for distributional equivalence, Standardized Mean Difference (SMD) for magnitude of difference, and Proportional Tolerance (15% margin) for categorical prevalence.

Key Contributions

• Generative Semantic Auditing: The paper introduces a scalable framework that automates the creation of semantic unit tests tailored to specific cohorts and datasets, moving beyond static rule libraries.
• Semantic Gap Identification: The methodology successfully identifies discrepancies where data passes syntactic checks but fails epidemiological plausibility checks.
• Reference Implementation (MDPT): A working tool utilizing a RAG architecture to synthesize medical literature into executable code, featuring a self-correcting "Auditor Agent" to enhance grounding accuracy.
• Three-Level Validation Taxonomy: A structured approach distinguishing between metadata, distributional, and contextual semantic layers.

Results
The authors evaluated MDPT across four clinical cohorts: Type 2 Diabetes (T2D) and Chronic Kidney Disease (CKD) in the All of Us (AoU) dataset, Hypertension (HTN) in SyntheticMass, and Congestive Heart Failure (CHF) in MIMIC-III.

• Test Generation: The framework autonomously generated between 55 and 73 unit tests per cohort. The Auditor Agent corrected approximately 39–55% of initial tests to match clinical references, discarding only 2–10%.
• Reference Validity: The validity of literature-derived expected values ranged from 35% to 93% across cohorts. When excluding drug prevalence tests (which suffer from data scarcity), validity increased to 39–98%.
• Semantic Divergence: The framework identified substantial divergence across all datasets. In cohorts where tests were verified as "correct" (grounded in reliable evidence), failure rates were high: 87.1% for CHF, 97.3% for T2D, and 100% for HTN and CKD. This indicates that syntactically valid data often fails to align with general population epidemiological priors.
• Bias Detection: As a positive control, MDPT correctly flagged demographic anomalies in the AoU dataset (e.g., over-representation of women) without manual configuration.
• Formatting Errors: In the MIMIC-III CHF cohort, the system detected a systematic formatting issue (omission of decimal points in ICD-9 codes) that rendered valid data semantically invisible to standard phenotype algorithms, causing observed prevalence to drop to near zero.
• Sensitivity Analysis: Failure rates were found to be more sensitive to proportional tolerance thresholds than to SMD cutoffs.

Significance and Claims
The paper positions Medical Data Pecking as a necessary evolution in data quality assurance, shifting from manual, context-agnostic rules to generative, context-sensitive verification. The authors claim that:

1. Syntactic completeness is insufficient: Data can be structurally perfect yet semantically implausible; verification against external evidence is required to reveal clinically relevant mismatches.
2. Scalability: The generative approach overcomes the scalability barriers of manual rule definition, enabling automated semantic auditing for diverse datasets.
3. Distinction between Error and Bias: High failure rates do not necessarily imply corrupted data; rather, they quantify the distance between a specific study cohort (e.g., ICU patients) and general population baselines. This helps researchers distinguish between technical data errors (e.g., formatting issues) and valid cohort-selection effects.
4. Operationalizing Policy: The framework offers a mechanism to operationalize high-level policy recommendations regarding data quality and algorithmic bias into executable code, addressing the "Garbage in - Garbage out" concern in medical AI.

The authors remain modest regarding limitations, acknowledging that reliance on general population priors can introduce a "reverse ecological fallacy" where valid clinical divergences in specialized cohorts are flagged as errors. They also note that the current implementation relies on general benchmarks, and future iterations could incorporate local health system baselines to serve as ground truth. The work concludes that while testing can show the presence of bugs, it cannot prove their absence, but it provides a critical initial framework for scalable semantic auditing.