Neural Signals Generate Clinical Notes in the Wild

This paper introduces CELM, the first foundation model for clinical EEG-to-language generation that leverages a large-scale dataset of 9,922 reports and 11,000 hours of recordings to achieve significant improvements in summarizing long-term EEG data and generating comprehensive clinical reports.

The Gist

Imagine you are a doctor who spends hours staring at a massive, scrolling tape of squiggly lines. This tape is an EEG recording, a snapshot of a patient's brain waves. It might be hours long, filled with tiny spikes, dips, and rhythms that tell a story about seizures, sleep, or other neurological issues.

Right now, a human doctor has to manually watch this entire tape, spot the important moments, and then type up a long, detailed medical report. It's tedious, slow, and requires years of specialized training. If the doctor misses a tiny spike in the middle of a 4-hour recording, it could be a big problem.

This paper introduces a new AI assistant called CELM (Clinical EEG Language Model) that acts like a super-powered, tireless medical scribe. Here's how it works, broken down into simple concepts:

1. The Problem: Too Much Data, Too Little Time

Think of a 2-hour EEG recording like a 100-mile-long novel written in a secret code.

• The Old Way: Previous AI tools tried to read this novel by looking at just one sentence at a time (a few seconds of brain waves) or by trying to guess the plot based on a few keywords. They often got lost, missed the big picture, or couldn't handle the sheer length of the story.
• The Limitation: They were like trying to understand a whole movie by looking at a single frozen frame.

2. The Solution: CELM's "Smart Summarizer"

CELM is the first AI designed to read the entire novel and write the summary for you. It does this in three clever steps:

Step A: The "Chapter Summarizer" (Epoch-Aggregated Tokenization)

Imagine the 100-mile novel is too long to fit on a single piece of paper.

• What CELM does: Instead of trying to read every single letter, it breaks the story into "chapters" (10-second chunks). It reads each chapter, writes a tiny, perfect summary note for it, and then stacks those notes together.
• The Result: Instead of a 100-mile book, the AI now has a neat 10-page outline that captures the essence of the whole story without losing the important details. This solves the problem of the AI getting "too full" to read the whole thing.

Step B: The "Plot Connector" (Sequence-Aware Alignment)

A story isn't just a list of chapters; the chapters happen in a specific order, and what happens in Chapter 3 might depend on Chapter 1.

• What CELM does: Older AI tools just looked at the chapter notes and guessed the ending. CELM uses a special "glue" (Sequence-Aware Alignment) to understand the flow and timing. It knows that a spike in the brain waves now might be related to a seizure that happened ten minutes ago. It connects the dots across time, just like a detective piecing together a timeline.

Step C: The "Doctor's Voice" (Prompt Fusion)

Now that the AI has the story and the timeline, it needs to write the report.

• What CELM does: It acts like a junior doctor who has studied under a master. It takes the "chapter notes" and the "timeline," mixes them with any extra info the human doctor provides (like "This patient has a history of epilepsy"), and writes the final report in the exact style doctors expect. It doesn't just guess; it synthesizes the data into a coherent narrative.

3. The Results: A Game Changer

The researchers tested this new AI against existing tools and human-level baselines.

• The "Zero-Context" Test: They asked the AI to write a report without giving it any patient history, forcing it to rely only on the brain wave tape.
  • Old AI: Got about 20% of the story right (like a student guessing on a test).
  • CELM: Got about 50% right. That's a massive jump, meaning it actually understood the brain waves, not just the text.
• With Patient History: When given the patient's background, CELM improved even further, getting up to 95% better than the old methods.

Why This Matters

Think of this as moving from a typewriter to a smart assistant.

• Before: Doctors had to manually transcribe hours of brain data, risking fatigue and errors.
• Now: CELM can read hours of data in seconds, highlight the critical moments, and draft a professional report.

This doesn't replace the doctor; it gives them a superpower. It frees them from the boring, repetitive work of typing up reports so they can focus on what they do best: making the final diagnosis and caring for the patient. The paper also released the "recipe" (the code and data) so other scientists can build on this, potentially leading to faster diagnoses for epilepsy and sleep disorders in the future.

Technical Summary

Here is a detailed technical summary of the paper "Neural Signals Generate Clinical Notes in the Wild" (CELM).

1. Problem Statement

The generation of clinical EEG reports from long-term, high-resolution EEG recordings is currently a labor-intensive process requiring expert neurologists to visually inspect signals, identify abnormal patterns (phenotypes), and compose structured narratives. Existing automated approaches suffer from three critical limitations:

• Conceptual Misalignment: Many methods treat report generation as a two-step process (phenotype classification followed by text decoding), creating a bottleneck where intermediate classification errors propagate to the final report. They fail to model the task as an end-to-end generation problem.
• Methodological Constraints: Current models operate on short, fixed-length EEG segments and rely on rigid templates. They cannot capture the long-term temporal context (hours to days) or global diagnostic reasoning required for interpreting continuous EEG data.
• Practical Limitations: Existing tools are often task-specific (e.g., generating only "impressions") and lack the flexibility to produce multi-granularity reports (e.g., background activity, seizure events, and detailed descriptions) simultaneously.

2. Methodology: CELM (Clinical EEG Language Model)

The authors propose CELM, the first clinical EEG-to-Language foundation model designed for end-to-end report generation from raw, long-duration EEG signals. The framework integrates pretrained EEG foundation models with Large Language Models (LLMs) through three core components:

A. Data Curation & Benchmark Construction

• Dataset: A large-scale dataset was curated from the Harvard Electroencephalography Database (v4.1), comprising 9,922 clinical reports paired with ~11,000 hours of EEG recordings from 9,048 patients.
• Pipeline: An automated pipeline was developed to extract, structure, and normalize unstructured clinical notes into standardized sections (e.g., Reason for Study, EEG Description, Epileptiform Abnormalities, Impressions).
• Preprocessing: EEG signals were band-pass filtered (0.1–75 Hz), resampled to 200 Hz, and standardized to a 22-channel montage.

B. Model Architecture

CELM addresses the challenge of fitting massive EEG data into LLM context windows via three stages:

1. Epoch-Aggregated Tokenization (EAT):

   • Challenge: A typical 2-hour EEG recording contains ~31.7 million data points, far exceeding LLM context limits.
   • Solution: The model uses a pretrained EEG encoder (e.g., CBraMod) to tokenize 1-second mini-windows within 10-second epochs. These are then aggregated (via pooling or [CLS] tokens) into a single token per epoch. This compresses the sequence by ~220x, making long-duration recordings feasible for LLMs.

2. Sequence-Aware Alignment:

   • Challenge: Simple linear projection (used in vision-language models) fails to preserve the critical long-range temporal dependencies inherent in sequential EEG data.
   • Solution: Two strategies are proposed:
     • Sequence Context Alignment (SCA): Uses a lightweight linear-attention transformer to capture temporal structure across epochs before projecting into the LLM embedding space.
     • Sequence Context Compression (SCC): Inspired by Perceiver architectures, this uses learnable query tokens to attend to the full sequence via cross-attention, compressing variable-length EEG sequences into a fixed-length latent representation for memory efficiency.

3. Prompt Fusion and Generation:

   • The aligned EEG tokens are concatenated with special tokens ([EEG_START], [EEG_END]) and optional clinical context (patient history, reason for study).
   • A local LLM (Qwen3-4B) is fine-tuned using a next-token prediction objective to generate structured JSON reports containing specific sections (Description, Background, Abnormalities, Impressions).
   • Training Strategy: The EEG encoder and LLM backbone are frozen; only the alignment module is trained, ensuring efficient adaptation while preserving pretrained representations.

3. Key Contributions

1. First Clinical EEG-to-Language Foundation Model: CELM is the first model to perform end-to-end generation of multi-scale clinical reports directly from raw EEG, bypassing intermediate phenotype classification.
2. Novel Architecture for Long-Context EEG: The introduction of Epoch-Aggregated Tokenization and Sequence-Aware Alignment solves the dual challenges of extreme data volume and the need to preserve long-range temporal dynamics.
3. Scalable Benchmark Pipeline: The authors release a pipeline to construct structured EEG-report benchmarks from raw hospital data, addressing the lack of public, high-quality paired datasets for this task.
4. Comprehensive Evaluation: The study evaluates performance across diverse report sections, hospital sites, and settings (with and without patient history).

4. Experimental Results

The model was evaluated on two hospital sites (S0001 and S0002) using standard generation metrics (BLEU-1, ROUGE-1, METEOR).

• Performance with Patient History:

  • CELM achieved 70%–95% relative improvements over the strongest baselines.
  • ROUGE-1 scores improved from 0.2–0.3 (baselines) to 0.4–0.6 (CELM).
  • Specifically, on site S0002, ROUGE-1 jumped from 0.2886 (best baseline) to 0.6408 (CELM).

• Zero-Context Performance (No Patient History):

  • In the absence of clinical history, CELM scored 0.43–0.52, significantly outperforming baselines (0.17–0.26). This demonstrates the model's ability to extract clinically relevant information solely from neural signals without hallucinating context.

• Ablation Studies:

  • Alignment: The Sequence-Aware Alignment (SCA) module significantly outperformed simple linear projectors and Perceiver projectors, confirming that modeling inter-epoch temporal dependencies is crucial.
  • Encoder: Using CBraMod as the EEG encoder yielded better results than LaBraM.
  • Compression: While the compressed variant (SCC) was memory-efficient, the non-compressed SCA variant achieved higher accuracy, highlighting a trade-off between efficiency and performance.

5. Significance and Future Implications

• Clinical Workflow Automation: CELM demonstrates the feasibility of automating the most time-consuming part of EEG analysis, potentially reducing the workload on neurologists and accelerating diagnosis.
• Multimodal Foundation Models: This work establishes a new paradigm for "Neural-to-Language" modeling, proving that foundation models can bridge the gap between raw physiological signals and complex clinical narratives.
• Limitations & Future Work:
  • Evaluation: Current metrics (ROUGE/BLEU) measure lexical similarity but not clinical correctness; future work needs clinically grounded evaluation protocols.
  • Scalability: While capable of handling ~3-hour recordings, further memory-efficient techniques are needed for ultra-long monitoring.
  • Rare Events: Performance was lower on "Interictal Epileptiform Abnormalities," suggesting a need for better modeling of rare, complex events.

In conclusion, CELM represents a significant leap forward in clinical neurophysiology AI, moving from rigid, template-based tools to flexible, end-to-end foundation models capable of generating comprehensive, multi-section clinical reports from raw neural data.