Comparing computable structured phenotype- versus large language model-identification of opioid use disorder using electronic health record data

This study demonstrates that while both a structured phenotype and a large language model effectively identify opioid use disorder in emergency department records, the LLM achieves significantly higher specificity and positive predictive value, suggesting it could reduce false-positive alerts in clinical workflows.

The Gist

Imagine the Emergency Department (ED) as a bustling, high-speed train station. Every day, thousands of people rush through, some with minor scrapes, others with life-threatening emergencies. Among this crowd are people struggling with Opioid Use Disorder (OUD)—a serious addiction that needs help. The problem is, the station staff (doctors) are so busy keeping the trains running that they often miss these specific passengers, or they accidentally flag innocent travelers as needing help, causing unnecessary panic and wasting time.

This study is like a test to see which security scanner works best at finding the right people without stopping the wrong ones. The researchers compared two different "scanners":

1. The Old Scanner: The "Rule-Based" System

Think of the Structured Phenotype as a very strict, old-school metal detector. It follows a rigid checklist:

• Did the person have a specific code in their file?
• Are they taking specific medications?
• Did a urine test come back positive?
• Did a doctor write a specific keyword like "heroin" or "fentanyl"?

If the person hits any of these boxes, the alarm goes off.

• The Good: It catches almost everyone who needs help (High Sensitivity).
• The Bad: It's a bit clumsy. It sometimes beeps for people who are just taking painkillers for a broken leg or had a one-time accident, even if they aren't addicted. This creates "false alarms" (False Positives), annoying the staff and making them ignore the real alerts later.

2. The New Scanner: The "AI Detective" (LLM)

The Large Language Model (LLM) is like a brilliant, super-fast detective who reads the entire story of the patient's visit, not just the checkboxes. It reads the doctor's handwritten notes, the conversation transcripts, and the context.

• The Good: It understands nuance. It can tell the difference between someone taking pain meds for surgery and someone with a deep addiction. It rarely makes mistakes on innocent people (High Specificity).
• The Bad: It missed a few people the old scanner caught, mostly because those people had their history written in a different hospital's notebook that the AI couldn't see.

The Race Results

The researchers let both scanners run on 302 real patients, then had two expert doctors (the "Human Judges") review the charts to see who was actually right.

• The Old Scanner (Rule-Based): Caught 84% of the real cases. But it falsely accused about 3.6% of innocent people.
• The New Scanner (AI): Caught 81% of the real cases (almost the same!). But it falsely accused only 0.4% of innocent people.

The Big Win: The AI was much better at not crying wolf. In a busy train station, if your alarm goes off every time someone walks by with a belt buckle, you eventually stop listening. The AI's ability to be precise means doctors can trust the alert more.

The "False Alarm" vs. "Missed Train" Analogy

• False Positives (The Old Scanner's flaw): Imagine the scanner beeping for a grandmother with a hip replacement who is taking painkillers. The doctor stops to investigate, wasting precious time, only to find she's fine.
• False Negatives (The AI's flaw): Imagine the scanner misses a passenger because their addiction history was written in a notebook from a different city that the scanner couldn't read.

The Verdict: A Team Effort

The study concludes that we shouldn't choose one over the other. Instead, we should use them as a tag-team:

1. First, use the Old Scanner to cast a wide net and make sure we don't miss anyone.
2. Then, let the AI Detective review the "alarms" to filter out the false ones.

This combination would act like a smart security system: it catches almost everyone who needs help but stops the noise that makes the staff tired and cynical.

The Catch

Just like any new technology, this AI needs to be tested in more train stations (hospitals) and with more diverse passengers. Also, the AI needs to be able to read notebooks from other cities (outside hospital records) to be perfect. But for now, it looks like a very promising tool to help doctors save lives without getting overwhelmed by false alarms.

Technical Summary

1. Problem Statement

Opioid Use Disorder (OUD) is a critical public health crisis, yet identifying patients in the Emergency Department (ED) is challenging due to time constraints, competing clinical priorities, and the limitations of current screening tools.

• Current State: Most EDs rely on computable structured phenotypes (rule-based systems using ICD codes, medications, lab results, and keywords) to trigger Clinical Decision Support (CDS) alerts. While efficient, these tools are often "blunt instruments" that may miss clinical nuance in narrative text or generate excessive false positives (e.g., flagging chronic pain patients on methadone).
• The Gap: The performance of Large Language Models (LLMs), specifically generative AI used without task-specific training (zero-shot), has not been rigorously tested against structured phenotypes for OUD identification in real-world ED settings. It is unclear if LLMs can better interpret narrative documentation to reduce false positives while maintaining sensitivity.

2. Methodology

The study employed a retrospective, single-center design at UCSF Medical Center involving adult ED encounters between January 1, 2023, and October 17, 2024.

• Study Population & Sampling:

  • Total pool: 70,880 adult encounters.
  • Stratified Sampling: To address the rarity of OUD, the authors used test-result-based sampling. They stratified encounters into two groups: those meeting a structured OUD phenotype and those that did not.
  • Final Sample: 302 unique patient encounters (201 phenotype-positive, 101 phenotype-negative).
  • Weighting: All statistical analyses applied inverse sampling weights to reflect the true population prevalence.

• Intervention 1: Structured Phenotype (The Comparator)

  • A rule-based algorithm currently used for CDS activation.
  • Criteria: Presence of ≥1 ICD-10 codes (OUD, withdrawal, overdose), active prescriptions for buprenorphine/methadone, positive urine toxicology (fentanyl, heroin, methadone), addiction consult notes, or specific keyword matches in ED notes.

• Intervention 2: Large Language Model (The Experimental)

  • Model: OpenAI ChatGPT 4.1.
  • Input: All ED notes from the index visit (excluding student/procedure notes) concatenated into a prompt.
  • Prompt Strategy: A zero-shot approach (no fine-tuning) iteratively refined over 10 test cases to determine OUD status based solely on clinical documentation.

• Reference Standard (Ground Truth):

  • Two board-certified emergency physicians (one with addiction/toxicology certification) independently reviewed the entire medical record, including external data via "Care Everywhere" (health information exchange).
  • Definition of OUD: Documented history of OUD, opioid withdrawal, intentional overdose, or >1 episode of intentional opioid use. (Opioid dependence was explicitly excluded).
  • Discrepancies were adjudicated by a third physician.

• Statistical Analysis:

  • Calculated weighted Sensitivity, Specificity, Positive Predictive Value (PPV), and Negative Predictive Value (NPV) with 95% Confidence Intervals (CI).
  • Used survey-weighted paired difference-in-proportions Wald tests to compare performance.

3. Key Contributions

• Direct Comparison: This is one of the first studies to directly compare a rule-based structured phenotype against a generative LLM (zero-shot) for OUD identification in an acute care setting.
• Rigorous Reference Standard: The use of expert human review with access to external records (Care Everywhere) establishes a high-fidelity ground truth, acknowledging that OUD documentation often resides outside the immediate EHR.
• Error Pattern Analysis: The study provides a qualitative breakdown of why each system fails (e.g., LLMs confusing dependence with disorder vs. structured phenotypes flagging chronic pain patients).

4. Results

• Prevalence: The weighted OUD prevalence in the population was 5.6% (95% CI 4.0–7.0%).
• Interrater Reliability: High agreement between human experts (Cohen's κ = 0.92).

Performance Metrics (Weighted):

Metric	Structured Phenotype	LLM (ChatGPT 4.1)	Statistical Significance
Sensitivity	0.84 (0.42–0.97)	0.81 (0.70–0.88)	$p = 0.859$ (No significant difference)
Specificity	0.964 (0.96–0.97)	0.996 (0.993–0.998)	$p < 0.0001$ (LLM significantly higher)
PPV	0.58 (0.51–0.64)	0.92 (0.85–0.96)	LLM significantly higher
NPV	0.99 (0.93–1.0)	0.99 (0.98–0.99)	Comparable

Qualitative Findings (Discordant Cases):

• Structured Phenotype False Positives: Primarily driven by single positive fentanyl urine toxicology results without evidence of misuse, or chronic pain patients on methadone/buprenorphine.
• Structured Phenotype False Negatives: Missed OUD history documented only in external records (Care Everywhere).
• LLM False Positives: Occasionally confused opioid dependence with OUD, or misinterpreted a single suicide attempt by overdose as chronic disorder.
• LLM False Negatives: Similar to the structured phenotype, missed OUD history located exclusively in external records or specific anesthesia notes.

5. Significance and Implications

• Reducing Alert Fatigue: The LLM demonstrated near-perfect specificity and a significantly higher PPV (92% vs. 58%). In low-prevalence settings like the ED, this suggests LLMs could drastically reduce the number of false-positive alerts, preventing "alert fatigue" among clinicians.
• Complementary Strengths: While the structured phenotype had slightly higher (though statistically non-significant) sensitivity, the LLM's ability to interpret narrative nuance makes it superior for precision.
• Proposed Workflow: The authors suggest a staged CDS approach:
  1. Use the Structured Phenotype as a high-sensitivity initial filter to cast a wide net.
  2. Pass positive cases to the LLM for a secondary review to filter out false positives and improve precision before triggering a clinical alert.
• Limitations & Future Work: The study was single-center, and the reference standard relied on external data access not available everywhere. Future work requires prospective validation across diverse health systems and the integration of longitudinal data to refine diagnostic criteria for AI.

Conclusion: Both methods are effective, but the LLM offers a distinct advantage in specificity and precision. Integrating LLMs into ED workflows could optimize OUD identification by balancing case detection with the reduction of unnecessary clinical interventions.