Automated epilepsy and seizure type phenotyping with pre-trained language models

This study demonstrates that a fine-tuned large language model (DeepSeek-R1) can accurately extract expert-level epilepsy and seizure phenotypes from unstructured clinical notes at scale, effectively transforming electronic health records into a valuable resource for longitudinal population research and improved patient care.

The Gist

Imagine you have a massive library containing millions of books about people with epilepsy. These aren't just dry statistics; they are the actual, handwritten (or typed) stories doctors wrote down during patient visits over the last 13 years.

The problem is that these stories are messy. The most important details—like exactly what kind of seizures a patient has or what specific type of epilepsy they suffer from—are buried deep inside paragraphs of text. Traditional computer systems are like librarians who can only read the library's index cards (structured codes). If a doctor didn't check a specific box on a form, the computer thinks the information doesn't exist, even though it's right there in the story.

This paper is about teaching a new kind of "super-librarian" to read those messy stories and extract the hidden gold.

The Two Competitors: The Specialist vs. The Genius

The researchers set up a race between two types of Artificial Intelligence (AI) to see who could read these doctor's notes best:

1. The Specialist (BERT): Think of this as a highly trained medical student who has studied thousands of similar notes. It's very good at following rules and patterns it has seen before, but it can get confused if the story is written in a weird way or uses unusual words.
2. The Genius (DeepSeek-R1): This is a Large Language Model (LLM). Imagine a brilliant, well-read professor who has read almost everything in the library. You don't need to train it on specific examples; you just ask it a question, and it uses its massive general knowledge to figure out the answer. It's like asking a human expert, "Based on this note, what kind of epilepsy does this patient have?"

The Test: Can the AI Think Like a Doctor?

To see if these AI "librarians" were any good, the researchers gave them a stack of 309 patient notes and asked them to identify two things:

• The Epilepsy Type: Is it "Focal" (starting in one spot), "Generalized" (starting everywhere), or "Unspecified"?
• The Seizure Type: Does the patient have "Convulsive" (shaking) seizures or "Non-convulsive" (staring/spacing out) seizures?

They compared the AI's answers to the answers given by a panel of real, board-certified epilepsy doctors.

The Results:

• The Specialist (BERT) did a decent job, but it struggled when the notes got complicated or ambiguous. It was like a student who memorized the textbook but got flustered by a tricky exam question.
• The Genius (DeepSeek) performed just as well as the human doctors, and in some tricky categories, it was even better than the humans. It understood the nuance of the language much better. It was like the professor who could spot the subtle clues in the story that the student missed.

The Big Reveal: Reading the Whole Library

Once they found the winner (DeepSeek), they didn't stop at 300 notes. They unleashed the AI on 77,000 notes from 18,500 patients. This is like going from reading a single chapter to reading the entire encyclopedia of epilepsy at their hospital.

Here is what the AI discovered by reading all those stories:

1. Diagnoses Evolve: At first, many patients get a vague diagnosis like "we think it's focal, but we're not sure." As time goes on and more tests are done, the AI saw that these vague labels often sharpen into specific, clear diagnoses. It's like a detective story where the suspect is initially just "a stranger," but by the end, it's "the butler with the candlestick."
2. Seizures Mix and Match: Many patients don't just have one type of seizure. The AI found that it's very common for a patient to have both "shaking" seizures and "staring" seizures, or even mix epileptic seizures with non-epileptic ones (like panic attacks that look like seizures). This complexity is hard to capture with simple checkboxes but easy for the AI to see in the text.
3. The "Generalized" Surprise: The AI confirmed that patients with "Generalized" epilepsy are much more likely to have the dangerous, full-body shaking seizures (tonic-clonic) compared to those with "Focal" epilepsy. This is crucial because those specific seizures are the biggest risk factor for a tragic event called SUDEP (Sudden Unexpected Death in Epilepsy).

Why This Matters

Think of this technology as a time machine for medical research.

Before, if a researcher wanted to study how epilepsy changes over 10 years, they had to hire a team of humans to read thousands of notes one by one. It would take years and cost a fortune.

Now, with this AI, we can instantly turn a mountain of messy, unorganized doctor's notes into a clean, organized database. This allows doctors to:

• Find patients who are at high risk for dangerous seizures much faster.
• See which treatments work best for specific types of epilepsy across huge groups of people.
• Understand the "life story" of a patient's disease, rather than just a snapshot of one day.

In short, this paper shows that AI can finally read the "fine print" in our medical records, turning a chaotic library of stories into a powerful tool that helps doctors save lives and treat patients more effectively.

Technical Summary

1. Problem Statement

Epilepsy is a complex neurologic disorder requiring precise classification of epilepsy type (e.g., focal, generalized) and seizure type (e.g., convulsive, non-convulsive) to guide treatment, surgical evaluation, and prognosis. However, current Electronic Health Record (EHR) systems fail to capture these nuances effectively:

• Structured Data Limitations: Diagnostic codes (ICD) are often too broad or inaccurate to reflect the specific, granular phenotypes required for clinical research.
• Unstructured Data Burden: Critical phenotypic details are buried in unstructured clinical notes (progress notes, EEG reports), making them inaccessible for large-scale, longitudinal analysis.
• Research Bottleneck: Consequently, research cohorts rely on small, manually curated datasets, limiting the generalizability of findings regarding seizure freedom, treatment response, and risks like Sudden Unexpected Death in Epilepsy (SUDEP).

The study aims to bridge this gap by developing and validating automated Natural Language Processing (NLP) methods to extract expert-level epilepsy phenotypes from unstructured clinical text at a health-system scale.

2. Methodology

Data Collection and Annotation

• Dataset: 77,049 outpatient clinical notes from 18,566 unique patients at the University of Pennsylvania Epilepsy Center (2011–2024).
• Annotation: Three board-certified epileptologists annotated a subset of 309 notes using the INCEpTION platform.
• Labeling Schemes:
  • Epilepsy Type: Single-label classification (Ternary: Focal, Generalized, Unspecified; Six-way; All-way granular).
  • Seizure Type: Multi-label classification (Binary: Convulsive vs. Non-convulsive; Ternary; All-way granular).
• Ground Truth: Created via consensus adjudication of inter-annotator disagreements.

Model Architecture and Training

The authors evaluated two distinct transformer-based architectures:

1. Fine-tuned BERT: A masked language model.
   • Pre-training: Domain-adaptive pre-training on ~64,000 unannotated clinical notes.
   • Fine-tuning: Supervised fine-tuning on the expert-annotated dataset with cross-validation.
2. DeepSeek-R1 (LLM): An autoregressive Large Language Model (specifically DeepSeek-R1-Distill-Llama-8B).
   • Configuration: Evaluated using Zero-shot (instructions only) and Few-shot (instructions + examples) prompting.
   • Optimization: Prompts were refined via error analysis to handle ambiguous subtypes.
   • Ensembling: Multiple independent predictions were generated and ensembled to improve robustness.

Deployment and Analysis Pipeline

• Selection: DeepSeek-R1 was selected for full deployment based on superior benchmark performance.
• Patient-Level Phenotyping: A two-stage adjudication process was applied to visit-level predictions:
  1. Majority voting across visits to assign a ternary epilepsy category.
  2. Selection of the most specific subtype within that category.
• Longitudinal Modeling: A Continuous-Time Hidden Markov Model (CT-HMM) was used to analyze diagnostic transitions over time, accommodating irregular visit intervals.
• Outcome Correlation: Automated phenotypes were integrated with previously extracted seizure frequency and freedom data to analyze outcomes (e.g., tonic-clonic burden, seizure-free duration) across epilepsy types.

3. Key Results

Model Performance vs. Human Experts

Performance was measured using Matthews Correlation Coefficient (MCC) and F1 scores, benchmarked against human inter-rater agreement.

• Epilepsy Type Classification (Ternary):

  • DeepSeek (Zero-shot): MCC = 0.85 (95% CI: 0.80–0.90).
  • Human Agreement: MCC = 0.77.
  • BERT: MCC = 0.73.
  • Finding: DeepSeek outperformed both BERT and human inter-rater agreement.

• Seizure Type Classification (Binary):

  • DeepSeek (Few-shot): MCC = 0.74 (95% CI: 0.66–0.81).
  • Human Agreement: MCC = 0.49.
  • BERT: MCC = 0.60.
  • Finding: Both models exceeded human agreement, with DeepSeek showing a significant advantage.

• Granular/Multi-label Tasks:

  • DeepSeek maintained performance comparable to or exceeding human agreement in complex, multi-label seizure classification (F1 = 0.72).
  • BERT performance declined significantly in granular tasks and failed in multi-label settings (defaulting to single labels).

System-Level Clinical Patterns

Deploying the model on the full corpus (77k notes) revealed:

• Diagnostic Stability: Unspecified focal epilepsy and unclassified states showed high transition rates to other states, while "specified" focal and generalized epilepsies showed high self-transition probabilities (diagnostic stability).
• Seizure Co-occurrence: High overlap in seizure types within patient records (e.g., co-occurrence of tonic-clonic and non-convulsive seizures, and Psychogenic Non-Epileptic Seizures [PNES] with true epileptic events).
• Outcome Correlations:
  • Tonic-Clonic Burden: Patients with generalized epilepsy had a 4-fold higher odds of tonic-clonic seizures compared to focal epilepsy (OR = 3.97).
  • Seizure Frequency: Lennox-Gastaut Syndrome (LGS) patients showed markedly higher seizure burdens compared to other subtypes.
  • Prognosis: Maximum seizure-free duration was similar between focal and generalized epilepsy, challenging the assumption that generalized epilepsy always has a more favorable prognosis in this tertiary cohort.

4. Key Contributions

1. Scale: Created the largest automated phenotyping dataset for epilepsy to date (77,049 notes, 18,566 patients).
2. Performance Benchmarking: Demonstrated that autoregressive LLMs (DeepSeek-R1) can achieve expert-level or superior performance compared to both fine-tuned BERT models and human epileptologists in complex, multi-label clinical classification tasks.
3. Methodological Framework: Established a scalable pipeline for converting unstructured EHR text into longitudinal, population-informed phenotype data, including a robust adjudication strategy for patient-level classification.
4. Clinical Insights: Uncovered system-level patterns regarding diagnostic evolution (stabilization over time) and the complex co-occurrence of seizure types that were previously inaccessible via structured data.

5. Significance and Limitations

Significance:

• Research Enablement: Transforms unstructured EHR data into a resource for large-scale, longitudinal studies of epilepsy trajectories, treatment response, and SUDEP risk stratification.
• Clinical Utility: Enables automated pre-population of clinical registries and trial screening lists, reducing manual abstraction burdens.
• Precision Medicine: Facilitates the identification of candidates for surgical or device-based interventions by accurately characterizing seizure types at scale.

Limitations:

• Inference Latency: DeepSeek-R1 has substantially longer inference times than BERT, posing challenges for real-time clinical deployment.
• Generalizability: The model was trained and tested on data from a single Level IV academic epilepsy center; performance may vary across different institutions, documentation styles, or EHR systems.
• Ground Truth Uncertainty: Inter-annotator disagreement (e.g., Kappa ~0.50 for seizure types) indicates that the "ground truth" itself contains inherent clinical ambiguity.
• Prompt Sensitivity: Iterative prompt refinement for DeepSeek may have inflated performance estimates on the held-out test set.

In conclusion, this study validates that pre-trained language models can effectively extract high-fidelity epilepsy phenotypes from clinical text, offering a scalable solution to unlock the value of unstructured EHR data for both clinical care and translational research.