On Using Large Language Models to Enhance Clinically-Driven Missing Data Recovery Algorithms in Electronic Health Records

Technical Summary: Enhancing Clinically-Driven Missing Data Recovery in EHRs with LLMs

Problem Statement
Electronic Health Records (EHR) are foundational to Learning Health Systems (LHS) and the operationalization of computable phenotypes, yet they suffer from significant missingness and errors. In the context of the Allostatic Load Index (ALI)—a phenotype measuring physiological "wear and tear" across cardiovascular, metabolic, and inflammatory systems—critical biomarkers are frequently missing from extracted EHR data. This missingness is often "not at random," depending on clinical decision-making (e.g., whether a physician orders a specific test). While expert chart reviews (source document verification) can recover missing values by searching for auxiliary diagnoses (anchors) in patient charts, this approach is resource-intensive, time-consuming, and lacks scalability, limiting its application to small subsamples.

Methodology
The study proposes a scalable, algorithmic alternative to manual chart reviews for recovering missing ALI components. The methodology involves three primary stages:

1. Baseline Algorithm Development: The authors adapted a previously developed "roadmap" created by clinical experts. This roadmap maps specific ALI components (e.g., Hemoglobin A1c, C-Reactive Protein) to search terms that can be matched against International Classification of Diseases, 10th revision (ICD-10) codes. The algorithm merges patient ICD-10 codes with these search terms; if a match is found, the missing biomarker is imputed as "unhealthy."
2. LLM-Enhanced Roadmap Generation: To expand the search terms beyond the original expert list, the authors utilized Large Language Models (LLMs), specifically Google's Gemini-2.5-Flash, via the ellmer package in R. They tested two prompt strategies:
   • LLM (Baseline): The LLM generated search terms based solely on biomarker names and health thresholds without context.
   • LLM (Context): The LLM was provided with the original expert roadmap examples and asked to generate an exhaustive list of additional terms iteratively (20 iterations per component).
3. Clinical Adjudication: A final roadmap was created by having two clinicians independently review and adjudicate the search terms proposed by the LLM (Context) roadmap. This step aimed to filter out clinically irrelevant matches (e.g., "pulmonary hypertension" for a metabolic marker) and prevent "hallucinations," resulting in the "LLM (Context + Clinicians)" roadmap.

The study utilized a cohort of 1,000 adults from an outpatient primary care setting. A subset of 100 patients underwent manual expert chart reviews to serve as a "gold standard" (alloyed with the extracted EHR data) for validating the algorithmic approaches.

Key Results

• Recovery Performance: The algorithmic approaches successfully recovered missing data comparable to, and in some cases exceeding, the manual chart reviews.
  • The LLM (Context) roadmap recovered the highest volume of missing components (55/413 missing points, 13%), slightly outperforming the expert chart reviews (45/413, 11%) and the clinicians' original roadmap (51/413, 12%).
  • The LLM (Context + Clinicians) roadmap recovered 45 missing points (11%), matching the expert review count exactly.
  • The LLM (Baseline) roadmap performed poorly, recovering only 3 missing components (7%), demonstrating that context is crucial for LLM performance in this domain.
• Scalability: While expert reviews were limited to 100 patients (requiring ~100 hours of human effort), the algorithm was applied to the full 1,000-patient sample. This increased the median number of non-missing ALI components per patient from 6 (raw EHR) to 7 (algorithmic recovery) across the entire cohort.
• Specific Component Recovery: The algorithm was most effective for C-Reactive Protein (CRP), recovering 34% of missing values, and Homocysteine (14%). It did not recover Serum Albumin (no search terms defined) or Body Mass Index (rarely missing).
• Statistical Impact: When applied to the full sample, the algorithmically recovered ALI values showed a linear relationship with the raw EHR values but aligned more closely with the "validated" values derived from chart reviews. In logistic regression models predicting healthcare engagement, the "augmented analysis" (using algorithmic recovery) produced odds ratios and estimates closer to the chart-review-based model than the "naive analysis" (raw EHR data).

Significance and Claims
The paper claims that clinically-driven algorithms, when enhanced by LLMs and subsequently adjudicated by clinicians, offer a cost-effective and scalable method to reduce missingness in EHR data. The authors assert that this approach:

1. Mimics Expert Accuracy: It achieves recovery rates similar to manual chart reviews without the associated time and labor costs.
2. Enables Large-Scale Phenotyping: It allows for the application of computable phenotypes (like ALI) to large samples (e.g., 1,000+ patients) where manual review is infeasible.
3. Improves Data Quality Dimensions: The primary focus is on improving completeness. The authors note that extending this framework to address other dimensions of data quality, such as plausibility or correctness, is a promising future direction.
4. Maintains Privacy: By passing only the roadmap structure (search terms) to the LLM rather than patient data, the method avoids privacy concerns and regulatory constraints associated with sharing Protected Health Information (PHI).

The authors conclude that while general-purpose LLMs are practical for this task, future work could benefit from domain-specific model training on medical terminology to further refine the precision of diagnostic term matching.