NeuroNarrator: A Generalist EEG-to-Text Foundation Model for Clinical Interpretation via Spectro-Spatial Grounding and Temporal State-Space Reasoning

The paper introduces NeuroNarrator, the first generalist foundation model that translates EEG signals into precise clinical narratives by leveraging a new 160K-scale dataset (NeuroCorpus-160K) and a novel architecture combining spectro-spatial grounding with state-space temporal reasoning to bridge electrophysiological dynamics and interpretable medical reporting.

The Gist

Imagine you have a radio that picks up the brain's electrical signals (EEG). Right now, most computers that listen to this radio are like strict traffic cops. They only care about specific, pre-defined signals: "Is there a seizure? Yes/No." "Is the person sleeping? Yes/No." They give you a simple checklist, but they can't tell you what the seizure looked like, where it started, or how it changed over time.

NeuroNarrator is a new kind of computer brain that acts more like a skilled radio announcer or a storyteller. Instead of just shouting "Seizure!" or "Sleep!", it listens to the brain's electrical waves and writes a detailed, flowing story about what is happening inside the head.

Here is how it works, broken down with simple analogies:

1. The Problem: The "Blurry Photo" vs. The "High-Res Video"

• Old Way: Imagine looking at a blurry, 10-second photo of a storm. You can see it's raining, but you can't tell if the wind is picking up, if a tree is falling, or if the storm is moving north. Most old AI models look at the whole EEG recording like this blurry photo. They miss the tiny, important details that happen in just a few seconds.
• NeuroNarrator's Way: This model looks at the brain activity like a high-definition video. It zooms in on tiny 10-second clips. It doesn't just say "there is activity"; it says, "At the 3-second mark, a spike of energy appeared in the right front of the brain, and it slowly faded away."

2. The Secret Sauce: "The Map and the Music"

To understand the brain, you need two things at once:

• The Music (Time): How the signal changes second-by-second (the rhythm, the spikes).
• The Map (Space): Where on the head the signal is loudest (left side, right side, top, back).

The Analogy: Imagine a concert.

• If you only listen to the music (the audio file), you know the song is loud, but you don't know where the drummer is standing.
• If you only look at the map (a photo of the stage), you see where the drummer is, but you don't hear the beat.
• NeuroNarrator forces the computer to look at the music and the map simultaneously. It uses a special "contrastive" trick (like a matching game) to ensure the computer understands that a loud drum beat in the audio must match the drummer standing on the left side of the photo. This prevents the AI from getting confused.

3. The "Time Travel" Feature

The brain is not static; it's a moving story. A seizure doesn't just "happen"; it starts, grows, and then stops. A sleepy brain doesn't just "be sleepy"; it slowly drifts from alert to drowsy.

• Old AI: Looks at the current moment like a snapshot. "What is happening right now?"
• NeuroNarrator: Uses State-Space Reasoning. Think of this as the AI having a short-term memory. Before it writes the story for the current second, it looks at the last few seconds (the "historical trajectory").
  • Analogy: If you see a car swerving, you don't just say "Car is moving." You say, "The car was driving straight, then it started to swerve left, and now it's hitting the curb." NeuroNarrator understands the story of the brain state, not just the snapshot.

4. The Massive Library (NeuroCorpus-160K)

To teach this AI to be a good storyteller, the researchers built a giant library called NeuroCorpus-160K.

• They gathered 16 different types of brain data (from sleep studies, epilepsy patients, people doing math, people feeling emotions, etc.).
• They didn't just label them "Good" or "Bad." They used a super-smart AI (GPT-4) to write human-like clinical descriptions for over 160,000 brain clips.
• Now, NeuroNarrator has read this library and learned the "vocabulary" doctors use to describe brain waves.

5. What Does It Actually Do?

When you feed NeuroNarrator a 10-second brain clip, it doesn't give you a number. It gives you a paragraph of text, like this:

"In this 10-second segment from a 19-year-old patient, we see a sudden spike in brain waves in the right front area. The energy is highest in the 'theta' rhythm, which is increasing compared to the previous few seconds. This suggests the brain is becoming drowsy."

Why Is This a Big Deal?

• For Doctors: It acts like a super-fast assistant. Instead of staring at a screen for hours, the doctor gets a draft report that highlights exactly what to look at. It saves time and reduces errors.
• For Science: It moves us away from "Yes/No" answers to rich, open-ended descriptions. It allows us to discover new patterns in brain data that we didn't even know to look for because we were too busy checking boxes.

In short: NeuroNarrator turns the chaotic, invisible static of brain electricity into a clear, coherent story that doctors can read and understand, bridging the gap between raw data and human insight.

Technical Summary

1. Problem Statement

Current computational approaches to Electroencephalography (EEG) analysis are largely limited to:

• Task-specific classification: Models are trained for narrow objectives (e.g., seizure detection, sleep staging) rather than open-ended interpretation.
• Coarse-grained analysis: Existing EEG-to-text methods often operate at the recording level, failing to capture transient, sparse, and fine-grained temporal events critical for clinical diagnosis.
• Lack of semantic grounding: Deep learning models typically encode EEG as abstract numerical time-series, disconnecting them from the open-ended, linguistically grounded reasoning required by clinical experts.
• Absence of temporal context: Most models treat EEG segments as independent snapshots, ignoring the non-stationary, evolving nature of brain states (e.g., seizure propagation or drowsiness onset) which relies on historical trajectory.

The authors argue that clinical EEG interpretation requires segment-level, open-vocabulary generation that explicitly preserves waveform morphology, spectral structure, spatial topography, and temporal evolution.

2. Methodology

The proposed framework, NeuroNarrator, is a Multimodal Large Language Model (MLLM) designed to translate electrophysiological segments into precise clinical narratives. The methodology consists of three core pillars:

A. Data Curation: NeuroCorpus-160K

• Scale & Diversity: The authors constructed the first large-scale, harmonized EEG-text corpus containing 160,249 training segments and 13,714 test segments derived from 16 heterogeneous public datasets (covering epilepsy, sleep, cognitive states, disorders, etc.).
• Subject-Disjoint Split: To ensure rigorous generalization, the data is split at the subject level, preventing data leakage between training and testing.
• Narrative Generation Pipeline:
  1. Preprocessing: Signals are standardized (0.1–75 Hz band-pass, notch filters, resampled to 200 Hz, harmonized electrode montages).
  2. Structured Feature Extraction: Each 10-second segment is converted into a quantitative template containing:
     • Event/Clinical Labels: Demographics and diagnosis.
     • Frequency-Domain Features: Power spectral density (PSD) across canonical bands.
     • Spatial Energy Distribution: Channel-wise power clustered into intensity tiers.
  3. Refinement: A domain-specific LLM (GPT-4.1) refines these rigid templates into fluent, clinically coherent narratives, incorporating implicit temporal context from preceding segments to describe evolving trends.

B. Architecture: Spectro-Spatial Grounding

The model employs a dual-stream encoding mechanism to align temporal waveforms with spatial topography:

1. Temporal Stream: Uses a pre-trained EEG encoder (LaBraM-Base) to process multichannel waveforms ($x_t$), extracting temporal dependencies.
2. Spatial Stream: Generates a scalp topographic map ($i_t$) for each segment and processes it using a frozen CLIP ViT-Large vision encoder. This anchors spatial features in a semantically rich manifold.
3. Contrastive Alignment: A contrastive learning objective (inspired by SigLIP) projects both streams into a shared latent space. It enforces strict correspondence between the temporal dynamics of the EEG and the spatial energy distribution of the topographic map, resolving grounding ambiguity.

C. Generation: State-Space Conditioned Decoding

• Input Construction: The model does not process segments in isolation. It constructs a composite input sequence ($E_{in}$) that includes:
  • Historical Context: Embeddings of preceding $N$ segments ($x_{t-N} \dots x_{t-1}$) to model the latent trajectory of brain states.
  • Current Evidence: Aligned spectro-spatial embeddings of the current segment ($x_t$ and $i_t$).
  • Instruction: Task-specific prompts.
• Mechanism: These continuous physiological embeddings are injected as "soft prompts" into a Qwen3-4B-Instruct language backbone.
• Objective: The model is trained end-to-end to minimize the negative log-likelihood of the target clinical narrative, effectively learning to synthesize evolving brain dynamics into coherent text.

3. Key Contributions

1. NeuroCorpus-160K: The first large-scale, open-vocabulary EEG-clinical narrative corpus with subject-disjoint splits, enabling benchmarking beyond closed-set classification.
2. Spectro-Spatial Alignment: A novel contrastive mechanism that rigorously aligns temporal EEG waveforms with spatial topographic maps, ensuring the generated text is grounded in both spectral and spatial evidence.
3. State-Space Reasoning Framework: The first generalist EEG-to-text model that conditions generation on historical trajectories, allowing it to capture non-stationary dynamics (e.g., seizure evolution) rather than static snapshots.
4. Open-Vocabulary Generation: Shifts the paradigm from fixed-label classification to free-form, clinically grounded narrative generation that supports compositional generalization across diverse tasks.

4. Experimental Results

• Spectro-Spatial Retrieval: The contrastive alignment achieved high retrieval accuracy (Average R@1: 84.19% for EEG-to-Map, 86.09% for Map-to-EEG), confirming the model successfully learned the correspondence between temporal and spatial features.
• Narrative Fidelity:
  • Lexical Metrics: Achieved an average BERTScore of 0.731 and Fact-F1 of 0.703, indicating strong semantic alignment and factual consistency with ground-truth reports.
  • Clinical Adjudication: Evaluated via a structured GPT-4.1 adjudicator across 5 axes (Event ID, Localization, Frequency, Trend, Non-dominant detection). The model scored highly, with human expert validation confirming the reliability of the automated scoring.
• Generalization (Zero-Shot): The model demonstrated robust zero-shot transfer to three external datasets (Depression-122, Siena, SEED-IV) without fine-tuning, achieving balanced accuracies of 52.94%, 62.53%, and 43.04% respectively.
• Comparison with Baselines: While specialized closed-set classifiers (e.g., EEGNet, EEGConformer) outperformed NeuroNarrator on specific label-matching tasks (due to task-specific optimization), NeuroNarrator provided superior interpretative fidelity and flexibility. It maintained performance across all benchmarks as a single unified model, whereas baselines required individual fine-tuning.
• Ablation Studies: Removing historical context caused the most significant performance degradation, validating the necessity of state-space reasoning for capturing non-stationary dynamics. Removing contrastive alignment or topographic maps also reduced fidelity, confirming the value of multimodal grounding.

5. Significance and Impact

• Clinical Workflow Integration: NeuroNarrator offers a tool to automate the drafting of clinical reports, directing expert attention to suspicious epochs and standardizing documentation, thereby reducing reporting burdens.
• Paradigm Shift: It moves EEG analysis from "black-box" classification to white-box, signal-grounded interpretation, making the reasoning process transparent and linguistically accessible.
• Foundation for Future Research: By establishing a framework for segment-level, open-vocabulary generation, this work paves the way for more complex temporal reasoning, integration of multi-modal clinical data, and real-time adaptive interpretation systems.
• Addressing Non-Stationarity: The explicit modeling of temporal trajectories addresses a fundamental limitation in current deep learning EEG models, aligning computational approaches more closely with the physiological reality of brain dynamics.