Technical Summary: Revealing Treatment Non-Adherence Bias in Clinical Machine Learning Using Large Language Models

Problem Statement

Machine learning (ML) systems trained on Electronic Health Records (EHRs) are increasingly utilized to guide clinical treatment decisions. These systems operate under a critical, often implicit assumption: that patients adhere to the treatments prescribed in their records. However, treatment non-adherence is a pervasive challenge in healthcare, particularly for chronic conditions like hypertension, where 40–50% of patients fail to take medications as prescribed.

Current methods for measuring adherence rely heavily on self-reported surveys and interviews, which suffer from small sample sizes and self-reporting bias. Physical monitoring solutions (e.g., electronic pill caps) are often impractical for large-scale deployment due to cost. Consequently, ML models trained on EHRs that ignore non-adherence learn patterns that misrepresent true treatment effects. This introduces an implicit bias that can distort causal inference, degrade predictive performance, and disproportionately exacerbate disparities in decision outcomes for vulnerable populations.

Methodology

Cohort Selection and Data

The study utilizes de-identified EHR data from the University of California, San Francisco (UCSF).

• Cohort: 3,623 patients with primary hypertension.
• Data Structure: Patients were paired based on consecutive primary care visits. The first visit involved a hypertension medication prescription; the second visit (occurring 1–12 months later) provided clinical notes for adherence verification.
• Medications: Focused on ten commonly prescribed antihypertensives (e.g., amlodipine, lisinopril, losartan).
• Features: Extracted demographic data (sex, age, race, marital status) and clinical factors (visit duration, hypertension duration, comorbidity counts via Charlson and Elixhauser indices).

Large Language Model (LLM) Extraction

To overcome the limitations of self-reporting, the authors employed a Large Language Model (GPT-4o) to extract adherence information directly from unstructured clinical notes.

• Input: For each patient pair, the model received the prescription record from the first visit and the clinical notes from the second visit.
• Prompting Strategy: A zero-shot approach was used. The model was prompted to identify instances of non-adherence, classify the type (e.g., missed dose, wrong dosage), and extract relevant text.
• Validation: A self-validation step was implemented where the model re-evaluated its initial output to reduce hallucinations.
• Human Validation: A subset of 100 notes (50 labeled non-adherent, 50 adherent by the model) was independently annotated by clinicians. The LLM achieved 92% accuracy, precision, and recall against this gold standard.

Analysis and Modeling

1. Statistical Analysis: Unadjusted and multivariate logistic regression were performed to identify factors associated with non-adherence.
2. Topic Modeling: BERT-based topic modeling (using UMAP for dimensionality reduction and HDBSCAN for clustering) was applied to the extracted non-adherent notes to uncover patient-reported reasons for non-adherence.
3. Impact Assessment:
   • Causal Inference: Inverse Probability Weighting (IPW) and Meta-learners (S-learner, T-learner, X-learner) were used to estimate the Average Treatment Effect (ATE). The study compared ATE estimates using the full dataset versus a dataset excluding non-adherent patients.
   • Predictive Modeling: Predictive models were trained to assess how ignoring non-adherence bias affects performance metrics and fairness across demographic groups.

Key Contributions

1. Large-Scale EHR Analysis: Conducted the first large-scale analysis of hypertension treatment non-adherence using EHR data (N=3,623), significantly larger than previous survey-based studies.
2. LLM-Based Extraction: Demonstrated that LLMs can effectively identify treatment non-adherence from clinical notes with 92% accuracy compared to physician annotations, offering a scalable alternative to self-reporting.
3. Identification of Drivers: Identified statistically significant factors associated with non-adherence (younger age, Black race, fewer comorbidities) and extracted specific patient-reported reasons (side effects, forgetfulness, refill difficulties).
4. Bias Quantification: Quantified the detrimental impact of ignoring non-adherence on both causal inference (reversing treatment effect estimates) and predictive modeling (degrading performance by up to 5% and worsening disparities).

Results

Adherence Statistics and Factors

• Prevalence: 786 patients (21.7%) were identified as non-adherent.
• Types of Non-Adherence: 64.4% missed treatments entirely; 30.2% took incorrect dosages; 6.7% used different medications; 7.9% took medication at incorrect times.
• Significant Predictors:
  • Age: Younger patients were less likely to adhere (p=0.036).
  • Race: Black patients exhibited higher rates of non-adherence (OR 1.38, p=0.016 in multivariate analysis).
  • Comorbidities: A lower number of comorbidities was associated with higher non-adherence rates (p=0.03).
• Reasons for Non-Adherence: Topic modeling revealed that side effects (dizziness, headaches) were the dominant reason (59.8%), followed by forgetfulness (17.7%), and logistical issues like needing refills or losing medication (11.6% and 11%, respectively).

Impact on Downstream Performance

• Clinical Outcomes: Non-adherence led to significantly worse blood pressure outcomes. Non-adherent patients showed 1.96 mmHg less systolic reduction (p=0.011) and 3.93 mmHg less diastolic reduction (p=0.001) compared to adherent patients.
• Causal Inference: Ignoring non-adherence bias caused the estimated treatment effects to be reversed or significantly distorted. Models trained on data that conflated non-adherence with treatment failure failed to capture the true efficacy of the medication.
• Predictive Performance: Including non-adherent patients without adjustment degraded model performance by up to 5%. Furthermore, this bias disproportionately affected vulnerable populations, exacerbating racial disparities in model error rates and decision outcomes.
• Mitigation: The study found that simply removing non-adherent patient records from the training dataset (despite reducing data volume) improved model performance and led to fairer predictions.

Significance and Claims

The paper argues that treatment non-adherence is not merely a clinical issue but a fundamental data quality problem for machine learning in healthcare. The authors claim that:

1. Implicit Bias is Critical: The assumption that prescribed treatments are taken is a source of implicit bias that fundamentally distorts both causal inference and predictive modeling.
2. LLMs are Viable Tools: LLMs provide a scalable, accurate method for extracting adherence data from unstructured clinical notes, enabling systematic assessment across large populations.
3. Equity Implications: Ignoring non-adherence does not just lower accuracy; it actively exacerbates health disparities by disproportionately affecting populations that already face barriers to adherence.
4. Actionable Insight: Addressing this bias is essential for developing responsible and equitable clinical ML systems. The authors suggest that filtering for adherence, even at the cost of dataset size, may be a necessary step to ensure model reliability and fairness.