🤖 machine learning

A foundation model with multi-variate parallel attention to generate neuronal activity

This paper introduces MVPFormer, a generative foundation model for human electrophysiology that utilizes a novel multi-variate parallel attention (MVPA) mechanism to effectively handle heterogeneous channel configurations, achieving state-of-the-art performance in seizure detection and decoding tasks while releasing the largest publicly available iEEG dataset, SWEC, to support future research.

Original authors: Francesco Carzaniga, Michael Hersche, Abu Sebastian, Kaspar Schindler, Abbas Rahimi

Published 2026-09-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Francesco Carzaniga, Michael Hersche, Abu Sebastian, Kaspar Schindler, Abbas Rahimi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The human brain is a vast, intricate network of electrical signals, constantly firing and communicating to sustain thought, movement, and sensation. To understand how this machinery works, and to detect when it malfunctions, doctors often place tiny electrodes directly inside the skull. This procedure, known as intracranial electroencephalography, or iEEG, captures the raw electrical activity of the brain with incredible precision. However, the data it produces is notoriously difficult to analyze. Unlike a standard heart monitor that uses a fixed set of sensors, iEEG setups are unique to every patient. A doctor might place twenty electrodes in one person's brain and eighty in another, depending on where the seizures are suspected to originate. This variability means that the number of channels and their specific locations change from person to person, making it nearly impossible for traditional computer models to learn from one patient and apply that knowledge to the next. For years, this lack of a universal way to read these shifting electrical maps has been a major barrier to developing smart tools that could help doctors diagnose and treat epilepsy more effectively.

A team of researchers has now built a new kind of artificial intelligence designed specifically to overcome this hurdle. They created a system called MVPFormer, which acts as a foundation model for human brain activity. Think of it as a general-purpose learner that can read the electrical language of the brain, regardless of how many sensors are attached or where they are placed. The core of this system is a new way of processing information called multi-variate parallel attention. Instead of trying to force every patient's unique electrode layout into a single, rigid grid, this method allows the computer to look at the data in three separate ways at once: it analyzes the content of the signal itself, the timing of the events, and the spatial relationship between the different channels. By separating these elements, the model can learn the underlying patterns of brain activity without needing to know the exact physical location of every electrode. It learns to recognize that a signal from one part of the brain might be connected to another, even if those parts are far apart, simply by observing how the electrical waves travel and interact.

To teach this system, the researchers needed a massive amount of data, but large, public datasets for this specific type of brain recording have been scarce. In response, they released the SWEC iEEG dataset, the largest collection of its kind ever made available to the public. This dataset contains nearly 10,000 hours of recordings from 68 different patients, gathered over a decade in clinical settings. It includes thousands of hours of normal brain activity as well as hundreds of recorded seizures. The researchers trained their model on a portion of this data, teaching it to predict what the brain's electrical signal would look like a few seconds into the future. This training process forced the model to understand the complex dynamics of how brain activity evolves over time. Once trained, they tested the model on patients it had never seen before, asking it to perform two critical tasks: first, to detect when a seizure was happening, and second, to decode specific cognitive tasks like recognizing speech or volume.

The results were striking. When tasked with identifying seizures in the new patients, the model performed at a level comparable to human experts. In the medical field, experts often disagree on the exact start and end of a seizure, but the model's agreement with the doctors was high enough to be considered expert-level performance. It successfully identified seizures across a wide variety of patients, even those with very different electrode setups and seizure patterns. Furthermore, the model outperformed existing state-of-the-art systems that rely on older methods of processing data. These older systems often failed completely when faced with the new, unseen patients, unable to generalize their knowledge. The new model, however, maintained its accuracy, proving that it had learned the fundamental rules of brain activity rather than just memorizing specific patient setups. It also excelled at decoding other brain functions, such as distinguishing between different volumes of sound or identifying when a person began to speak, surpassing previous models in these areas as well.

The researchers also tested whether this new approach worked outside the specific context of brain signals, applying it to standard time-series data like weather forecasts and energy consumption records. In these general tasks, the model matched or exceeded the performance of the best existing tools, suggesting that the method they developed is a powerful, flexible tool for analyzing any complex data that changes over time and involves multiple sources. Crucially, the team found that the model's ability to learn from the raw data was essential; a version of the model that was not first trained to predict future brain states performed significantly worse on the seizure detection task. This indicates that the initial phase of learning to anticipate the brain's next move was vital for building a system that could truly understand the complexity of human physiology.

By releasing both the model and the massive dataset to the public, the researchers have provided a new foundation for the field. They have demonstrated that it is possible to build an artificial intelligence that can navigate the messy, variable reality of clinical brain data without requiring a perfect, standardized setup for every single patient. The system does not need to know the exact map of a patient's electrodes to understand what is happening; it learns the connections and patterns on its own. This work suggests a future where computer models can assist doctors in real-time, offering reliable insights into brain activity across diverse patients, potentially leading to faster diagnoses and better management of neurological conditions. The success of this approach marks a significant step forward in turning the chaotic, patient-specific data of the brain into a coherent, understandable signal that machines can read and learn from.

Technical Summary: A Foundation Model with Multi-Variate Parallel Attention to Generate Neuronal Activity

Problem Statement

The paper addresses a fundamental challenge in deep learning for multi-variate time-series: channel heterogeneity. In domains like intracranial electroencephalography (iEEG), the number, location, and configuration of sensors (channels) vary significantly across subjects due to clinical needs. Existing models often require subject-specific adaptation or fail to generalize because they rely on fixed channel positions or global positional encodings that do not account for this variability. Furthermore, the lack of large-scale, publicly available iEEG datasets has hindered the development of foundation models capable of learning generalizable representations of brain activity.

Methodology

1. Multi-Variate Parallel Attention (MVPA)

The core technical contribution is Multi-Variate Parallel Attention (MVPA), a novel self-attention mechanism designed to disentangle content, temporal, and spatial dependencies without relying on fixed channel positions.

Decomposition: Unlike vanilla attention which treats the input as a flat sequence or dual-coded attention which computes expensive second-order correlations, MVPA decomposes the attention score into three additive components:
1. Content-based attention: Computes similarity based purely on the raw segment embeddings ( $x_{c,t}$ ), ignoring positional information.
2. Time-based attention: Depends only on the relative temporal distance between query and key ( $t - t'$ ), shared across all channels.
3. Channel-based attention: Depends only on the relative spatial distance between channels ( $c - c'$ ), shared across all time steps.
Relative Encoding: MVPA utilizes learnable bias terms and relative distance encodings (inspired by Transformer-XL) rather than absolute positional encodings. This allows the model to learn the hidden connection map between spatial locations dynamically, making it agnostic to specific electrode layouts.
Efficiency: To handle the quadratic complexity of 2D attention, the authors employ:
- Local windows for the content-based component (focusing on recent time segments).
- Shifting operations to compute relative embeddings in a single pass.
- Grouped Query Attention (GQA) and FlashMVPA (implemented in OpenAI Triton) to reduce memory usage and computational cost, enabling context lengths over 10,000 on standard GPUs.

2. MVPFormer Architecture

The authors introduce MVPFormer, a generative foundation model for human electrophysiology built upon the MVPA mechanism.

Architecture: Based on the Llama2 architecture, MVPFormer processes iEEG signals through a wavelet-based encoder (using db4 decomposition) to generate continuous embeddings. The decoder utilizes MVPA to model dependencies.
Training Strategy:
- Generative Pre-training: The model is trained to predict the evolution of neuronal activity by forecasting the next time segment in a continuous embedding space. This is achieved via a contrastive loss function that maximizes similarity to the true future target while minimizing similarity to "confounder" targets (random segments from the same batch).
- Fine-tuning: For downstream tasks, the model is fine-tuned using LoRA (Low-Rank Adaptation) on specific classification heads, requiring only ~0.1% of the parameters to be updated.

3. The SWEC iEEG Dataset

To support the training and evaluation of MVPFormer, the authors release the SWEC iEEG dataset.

Scale: It is the largest publicly available iEEG dataset to date, comprising 9,328 hours of recordings from 68 subjects and 704 ictal (seizure) events.
Characteristics: The data is heterogeneous, collected from various clinical sources with varying channel counts (ranging from 22 to 128 electrodes) and sampling rates. It is fully curated and labeled by experienced clinicians.

Key Results

1. Seizure Detection (SWEC, MAYO, FNUSA)

MVPFormer demonstrates expert-level performance in seizure detection across multiple datasets:

SWEC Dataset: MVPFormer achieves a Cohen's Kappa of 0.61 on 50 unseen subjects, matching human expert performance (typically 0.53–0.58). In contrast, state-of-the-art (SOTA) baselines like Brant-2, BrainBERT, and a vanilla-attention variant (MV-Llama) fail to generalize, achieving Kappa scores near 0.00–0.11.
Generalization: The model maintains strong performance on the MAYO and FNUSA datasets (which contain significant noise and fewer subjects), outperforming Brant-2 significantly on the noisy MAYO dataset (F1-score 0.36 vs. 0.19).
Ablation: A purely discriminative model trained without generative pre-training achieves a lower Kappa (0.52), validating the necessity of the generative pre-training phase for generalization.

2. Brain TreeBank Decoding Tasks

MVPFormer is evaluated on four high-level cognitive decoding tasks (volume, pitch, onset, speech) from the Brain TreeBank dataset.

It surpasses all vanilla-attention baselines (Brant, BrainBERT, MV-Llama) on the pitch and volume tasks.
It achieves SOTA performance on onset and speech tasks, outperforming models that explicitly utilize absolute electrode positions (PopT with encoding), demonstrating that MVPA's implicit channel map is sufficient and flexible.

3. General Time-Series Benchmarks

The authors validate MVPA on standard time-series forecasting (ETTh1, ETTh2, Weather) and classification tasks (EthanolConcentration, FaceDetection, etc.).

MVPA matches or exceeds the performance of SOTA models like PatchTST, TimesFM, and TimeMixer.
Ablation studies confirm that the full MVPA (content + time + channel components) outperforms variants using only single components, particularly on datasets with higher channel counts (e.g., Weather dataset).

Significance and Claims

The paper claims to establish MVPA as a general-purpose attention mechanism capable of handling heterogeneous multi-variate time-series without sacrificing locality or generalization.

The primary significance of the work lies in the realization of MVPFormer, described as the first open-source, open-weights, and open-data foundation model for iEEG with SOTA clinical performance. By releasing the SWEC dataset and the model weights, the authors aim to overcome the data scarcity barrier that has previously limited progress in iEEG foundation models. The results suggest that generative pre-training on large-scale, heterogeneous iEEG data enables models to learn robust representations of both physiological and pathological brain states, allowing for zero-shot generalization across diverse subjects and clinical setups where previous models failed.