Extracting patient reported cannabis use and reasons for use from electronic health records: a benchmarking study of large language models

This study demonstrates that combining fine-tuned clinical language models with large language model classifiers enables the high-precision, scalable extraction of patient-reported cannabis use status and reasons for use from unstructured electronic health records of patients with autoimmune rheumatic diseases.

The Gist

Imagine you are a detective trying to solve a mystery inside a massive library. This library contains millions of handwritten notes from doctors (Electronic Health Records) about patients with painful, chronic joint and immune diseases.

The mystery? Are these patients using cannabis to help with their pain, and if so, why?

The problem is that the library is chaotic. The notes aren't organized in neat boxes; they are scattered in messy paragraphs, full of slang, abbreviations, and hidden clues. A human detective would need years to read every single page. That's where this study comes in. They tried to teach Artificial Intelligence (AI) to become the ultimate detective.

Here is the story of their experiment, explained simply:

1. The Mission: Finding the Needle in the Haystack

The researchers wanted to build a system that could automatically scan these messy doctor's notes to find two things:

• Status: Is the patient using cannabis? (Yes, No, Used to, or Uncertain?)
• Motivation: If they are using it, why? (Is it for pain, sleep, nausea, or anxiety?)

2. The Tools: The "Smart" vs. The "Specialist"

To solve this, they tested two different types of AI detectives:

• The General Geniuses (Large Language Models): Think of these like Olympic-level general knowledge athletes. They are huge, incredibly smart, and can read almost anything. They are great at understanding complex stories and context. The researchers tested several of these (like GPT-OSS, Gemini, and Llama).
• The Specialized Interns (Fine-Tuned Clinical Models): Think of this like a medical student who has only studied one specific subject. They aren't as big or flashy as the geniuses, but they have been specifically trained on thousands of medical notes. They know exactly what a doctor means when they write "cannabis" in a specific way.

3. The Experiment: The Big Showdown

The researchers gave both types of AI a stack of 886 real doctor's notes and asked them to classify the cannabis use. Then, they gave them another stack of 1,027 notes to figure out the reasons for use.

The Results:

• For "Status" (Is it yes or no?): The Specialized Intern (GatorTron) won easily. It was like having a security guard who knows exactly what a "No Smoking" sign looks like. It was faster, cheaper to run, and made fewer mistakes than the big geniuses.
• For "Reasons" (Why are they using it?): The General Genius (GPT-OSS) took the crown. Figuring out why someone uses cannabis is tricky. Sometimes a doctor writes, "Patient seems less anxious," without explicitly saying "for anxiety." The big AI could read between the lines and understand the context better than the specialized intern.

4. The Temperature Setting: The "Calmness" Dial

The researchers also played with a setting called "Temperature."

• High Temperature: Imagine the AI is a jazz musician improvising wildly. It's creative but might make up facts or give inconsistent answers.
• Low Temperature: Imagine the AI is a robot following a strict script. It is boring but very reliable.
• The Finding: For medical notes, low temperature (being calm and strict) was always better. It made the AI more consistent and accurate.

5. What Did They Discover? (The Treasure Map)

Once they built the best system (using the Specialized Intern for status and the General Genius for reasons), they scanned the entire library of 2 million notes from 2015 to 2024. Here is what they found:

• The Trend is Up: Cannabis use among these patients has been steadily climbing, almost doubling from 2015 to 2024.
• The "Why": Pain is the number one reason people use it. However, starting in 2022, sleep problems became the second most common reason.
• The Pain Paradox: Interestingly, in the early years, patients using cannabis reported higher pain scores than those who didn't. But by 2023, this flipped, and non-users reported slightly higher pain. This suggests that as more people started using cannabis, the "pain gap" changed, though the study can't say exactly why (maybe the patients who used it were in more pain to begin with, or maybe the cannabis helped them manage it better over time).

The Big Takeaway

This study is like a blueprint for the future of medical research. It tells us:

1. Don't just use the biggest, most expensive AI for everything. Sometimes a smaller, specialized tool is better and cheaper.
2. Context matters. If you need to understand the story behind a note, you need the big, creative AI.
3. We can now listen to the "whispers" in the medical records. By using these AI tools, doctors and researchers can finally understand how real patients are managing their pain and what treatments are actually working in the real world, not just in clinical trials.

In short, they taught computers to read the messy notes of doctors, turning a chaotic library into a clear map of how patients are using cannabis to fight their pain.

Technical Summary

1. Problem Statement

Autoimmune rheumatic diseases (ARDs) are chronic conditions often managed with cannabis for pain and symptom relief. However, accurate data on cannabis use and the specific reasons for use (e.g., pain, sleep, anxiety) are embedded in unstructured Electronic Health Record (EHR) clinical notes. Traditional rule-based Natural Language Processing (NLP) approaches struggle with the variability of clinical documentation, requiring manual rule engineering that is difficult to scale. While Large Language Models (LLMs) offer a potential solution, there is a lack of empirical evidence regarding:

• The relative performance of general-domain LLMs (e.g., GPT, LLaMA) versus clinically pre-trained models (e.g., GatorTron, MedGemma) for this specific task.
• How prompting strategies (zero-shot vs. few-shot) and decoding temperatures affect extraction reliability.
• The feasibility of extracting not just status (use/denial) but also the context (reasons for use) at a population scale.

2. Methodology

Data Source and Study Design

• Dataset: Retrospective cohort of 5,580 adult patients with ARDs (e.g., RA, SLE, PsA) from a Northern California academic medical center (2015–2024).
• Volume: Over 2 million clinical notes.
• Preprocessing:
  • Snippet Extraction: Used fuzzy string matching (rapidfuzz library, similarity threshold ≥90) against a curated cannabis lexicon to identify relevant terms.
  • Context Window: Extracted 50-token windows (±25 tokens) around matches.
  • Normalization: Removed non-English characters but retained clinical symbols (e.g., colons, slashes) to preserve structured data patterns.
  • Result: 43,689 candidate snippets from 3,244 patients.

Annotation and Reference Standard

Two domain experts manually annotated two distinct datasets:

1. Cannabis Use Status (831 snippets): Four classes: (1) Not a true mention/uncertain, (2) Denial, (3) Positive past use, (4) Positive current use.
2. Reason for Use (1,027 snippets): Six classes: Pain, Nausea, Sleep, Anxiety/Stress/Mood, Appetite, and Not mentioned/unknown. (Note: High class imbalance, with "Not mentioned" being the majority).

Model Benchmarking

The study evaluated five models across different architectures:

• General-Domain Generative LLMs: Gemini-2.0, LLaMA-3.1-8B, GPT-OSS-20B.
• Clinical-Domain Models: MedGemma-4B, GatorTron (345M parameters).

Experimental Variables:

• Prompting Strategies: Eight strategies ranging from zero-shot to few-shot and structured reasoning (e.g., multi-step verification).
• Decoding Temperature: Tuned from 0.0 to 1.0 to assess the trade-off between creativity and reproducibility.
• Supervised Fine-Tuning: GatorTron was fine-tuned specifically for the "Status" classification task using Bayesian optimization (Optuna) on an 80/20 train/validation split.

Evaluation Metrics

Performance was measured on held-out test sets using Accuracy, Precision, Recall, and F1-score. Confidence intervals were generated via bootstrapping (100 iterations).

3. Key Results

A. Cannabis Use Status Classification

• Best Performer: The fine-tuned GatorTron model significantly outperformed all generative LLMs.
  • Metrics: Accuracy 0.90, F1 0.91, Recall 0.90, Precision 0.90.
• Generative Models: Among the prompt-based models, GPT-OSS-20B performed best but lagged behind GatorTron (Accuracy ~0.75).
• Temperature Effect: Lower temperatures (0.0–0.3) consistently yielded higher stability and accuracy across all generative models.
• Prompting: No single prompt strategy worked best across all models; performance was highly model-dependent.

B. Reason for Use Classification

• Best Performer: GPT-OSS-20B (using a multi-step verification prompt and temperature 0.3) achieved the highest performance.
  • Metrics: Accuracy 0.77, F1 0.77, Precision 0.86.
• Fine-Tuned Model Failure: GatorTron performed poorly on this task (F1 0.55), suggesting that the fine-tuned encoder model lacked the generative reasoning capability required for the semantically complex and heterogeneous "reason" categories.
• Insight: Generative LLMs are superior for high-entropy, context-rich extraction tasks, while fine-tuned discriminative models excel at structured classification.

C. Clinical Insights (Population Trends)

Applying the best models to the full cohort (2015–2024) revealed:

• Prevalence: Documented cannabis use increased steadily from 7.4% (2015) to 13.0% (2024).
• Demographics: Rheumatoid Arthritis (RA) patients had the highest prevalence; Ankylosing Spondylitis (AS) had the lowest.
• Reasons: Pain remained the dominant reason throughout the decade. Sleep emerged as the second most common reason starting in 2022.
• Pain Correlation: Prior to 2023, current users reported higher pain scores than non-users. This trend reversed in 2023–2024, where non-users reported slightly higher pain scores.

4. Key Contributions

1. Task-Specific Model Selection: The study establishes that there is no "one-size-fits-all" LLM for clinical NLP.
   • Structured Tasks (Status): Fine-tuned, smaller domain-specific models (GatorTron) are more efficient and accurate.
   • Unstructured/Contextual Tasks (Reasons): Large generative LLMs (GPT-OSS-20B) provide necessary contextual reasoning.
2. Benchmarking Framework: Provides empirical evidence that prompt engineering is not transferable across models and that lower decoding temperatures are critical for clinical reproducibility.
3. Scalable Phenotyping: Demonstrates a reproducible pipeline to extract patient-reported outcomes (PROs) from unstructured text at the population level, overcoming the limitations of manual chart review.

5. Significance and Limitations

Significance:

• Research: Enables large-scale observational studies on cannabis use patterns in ARDs, which were previously limited by the inability to process unstructured notes.
• Clinical Practice: Supports the development of clinical decision support tools that can identify patients using cannabis for self-management, potentially guiding pain management strategies.
• Efficiency: Highlights that for many clinical extraction tasks, fine-tuning a smaller model is more resource-efficient than running massive generative models.

Limitations:

• Documentation Bias: Relies on clinician documentation, which may underrepresent true prevalence due to stigma or variable recording practices.
• Context Window: The 50-token window may miss motivations described in distant parts of a note.
• Class Imbalance: The "Reason for Use" dataset had significant imbalance (e.g., "Not mentioned" was 61%), potentially affecting minority class sensitivity.
• Generalizability: The study was conducted in a single academic health system; external validation is required.
• Causality: The pain score analysis is cross-sectional; the study cannot infer causality between cannabis use and pain levels.

Conclusion

The paper concludes that a hybrid approach is optimal for clinical NLP: use fine-tuned discriminative models for well-defined, structured classification tasks to maximize efficiency, and deploy large generative LLMs for complex, context-dependent extraction tasks. This strategy balances computational cost with the high precision required for clinical phenotyping.