💬 NLP

End-to-End Intracortical Speech Decoding from Neural Activity

This paper demonstrates that an end-to-end Conformer-based neural decoder, trained directly on intracortical recordings from an ALS participant without external language models, achieves meaningful character-level speech decoding with a 23.80% character error rate, despite challenges from signal degradation and word boundary segmentation.

Original authors: Owais Mujtaba Khanday, Jose A. Gonzalez-Lopez, Marc Ouellet, Alberto Galdon, Gonzalo Olivares Granados

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Owais Mujtaba Khanday, Jose A. Gonzalez-Lopez, Marc Ouellet, Alberto Galdon, Gonzalo Olivares Granados

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine your brain is a bustling city with millions of tiny messengers (neurons) shouting out instructions. For most of us, when we want to speak, these messengers tell our lips and tongue what to do. But for people with conditions like ALS, the "wires" connecting the brain to the mouth are broken. The messengers are still shouting, but the message never reaches the lips.

This paper describes a new "translator" built to catch those shouts directly from the brain and turn them into text on a screen, without needing any help from outside dictionaries or grammar books.

Here is the story of how they built it, using simple analogies:

1. The Problem: The "Heavy Backpack"

Previous high-tech brain-to-text devices were like a student trying to solve a math problem while wearing a heavy backpack. The backpack represents an external language model (a massive database of how words usually go together).

The old way: The brain decoder guesses the letters, then the heavy backpack checks the guess, corrects it, and re-ranks the options.
The issue: This backpack is heavy (takes up lots of memory), slow (adds delay), and requires a powerful computer. For a device meant to be implanted inside a person's skull, carrying a "backpack" is impractical. The researchers asked: Can we build a translator that is so smart it doesn't need the backpack at all?

2. The Solution: The "Super-Translator" (The Conformer)

The team built a new kind of translator called a Conformer. Think of this as a highly trained detective who looks at the raw noise of the brain and figures out the story directly.

The Input: They recorded signals from a participant with ALS using tiny electrodes implanted in the part of the brain that controls speech.
The Goal: Instead of guessing whole words (which is hard), the system guesses letters one by one, directly from the brain waves.
The Result: Without using any external dictionary or grammar checker, this system got about 76% of the letters right (a 23.8% error rate). This proves that the brain signal itself is strong enough to be understood without the "backpack."

3. The Challenge: The "Shaky Camera"

One of the biggest hurdles in brain decoding is that the signal changes over time. Imagine trying to take a photo of a moving object with a camera that is slowly drifting, getting dirty, or changing its focus every day.

The Issue: The electrodes move slightly, or the brain's activity patterns shift from day to day. A model trained on Monday's data might fail on Tuesday's data.
The Fix: The researchers added a special "Session Adapter." Think of this as a pair of adjustable glasses the model puts on every morning. Before the main detective (the Conformer) looks at the data, these glasses quickly adjust the focus to match that specific day's "shaky camera." This allows the main detective to stay calm and consistent, even if the view changes.

4. The Training: "Practice with Noise"

To make the translator robust, they didn't just show it clean brain signals. They used a technique called Data Augmentation, which is like training a musician by playing music with static, speed changes, and missing notes.

They artificially added "static" (noise) to the signals.
They sped up and slowed down the recordings.
They randomly "blacked out" some of the electrodes (simulating a broken wire).
Why? This forced the model to learn the essence of the speech, not just memorize the specific noise of a single recording session. It's like teaching a driver to drive in rain, snow, and fog, so they can handle any weather on the road.

5. The Weakness: "Where do the words start and stop?"

While the system is great at guessing letters, it sometimes struggles with word boundaries.

The Analogy: Imagine reading a sentence where all the spaces are missing: HELLOHOWAREYOU. You know the letters, but you have to guess where one word ends and the next begins.
The Finding: The biggest mistakes the system made were either deleting a space (merging two words) or adding a space in the middle of a word. It wasn't usually confusing the letter "A" with "B"; it was confused about where the "space" should be.
Why? The brain signal for "space" is very faint compared to the signal for actual letters. It's like trying to hear a whisper in a loud room.

Summary

This paper shows that we can build a standalone, end-to-end translator that reads brain signals and turns them into text without needing a massive external computer to help fix the grammar.

Success: It works well enough to be a strong foundation for future devices.
Limitation: It still gets confused about where words begin and end (the "spaces"), and its performance drops a bit as the electrodes age or drift over time.
Takeaway: The brain's signal is clear enough to decode letters directly, paving the way for simpler, faster, and more implantable devices for people who cannot speak.

Technical Summary: End-to-End Intracortical Speech Decoding from Neural Activity

Problem Statement
Current high-performing intracortical speech neuroprostheses achieve low word error rates but typically rely on external language models (LMs) during inference to rescore outputs generated by Connectionist Temporal Classification (CTC) beam search. While effective for accuracy, this reliance introduces significant practical constraints: large vocabulary LMs demand substantial memory, and the sequential nature of beam search with LM integration creates latency proportional to beam width. These factors hinder the deployment of fully implanted, miniaturized, and power-efficient speech neuroprostheses that operate in real-time without tethered external computers. Furthermore, the role of external LMs is to impose linguistic structure rather than extract information directly from neural signals, raising the question of whether meaningful character-level decoding can be achieved in a fully end-to-end framework without external rescoring. Additionally, intracortical recordings suffer from inter-session non-stationarity due to electrode drift, impedance fluctuations, and changes in neural population dynamics, which degrade cross-session generalization.

Methodology
The authors propose a fully end-to-end neural decoder based on the Conformer architecture, trained directly on intracortical recordings from a participant with amyotrophic lateral sclerosis (ALS). The system maps neural activity to character sequences using a CTC objective without any external language model at inference time.

The proposed architecture consists of four key components:

Session-Specific Linear Alignment: To address inter-session non-stationarity, a lightweight, session-specific linear adapter layer precedes the shared encoder. This layer applies a session-specific weight matrix ( $W_d$ ) and bias vector ( $b_d$ ) with a SiLU activation to normalize feature distributions for each recording session. These parameters are optimized independently with a higher learning rate to allow rapid alignment while the shared encoder learns session-invariant representations.
Temporal Patch Embedding: Inspired by Vision Transformers, the model employs a strided temporal patch embedding to reduce sequence length before the encoder. This compresses temporal resolution, reducing the computational complexity of subsequent self-attention layers while preserving local temporal dynamics.
Conformer Encoder: The core encoder comprises six stacked Conformer blocks utilizing a "Macaron-style" sandwich architecture. Each block integrates multi-head self-attention (to capture global dependencies) and convolutional modules (to capture local articulatory patterns), sandwiched between feed-forward networks.
Character Prediction Head: A two-layer linear head predicts character probabilities over a vocabulary of 129 tokens (128 printable ASCII characters plus a blank token).

Training and Data Augmentation
The model is trained on the "Brain-to-Text '25" benchmark, comprising 10,948 trials from 45 recording sessions over 20 months. To enhance robustness against the inherent non-stationarity and low signal-to-noise ratio of intracortical data, the authors employ a targeted data augmentation strategy applied in the feature space:

Additive Noise: White noise, constant offset noise, and random walk noise simulate channel noise and baseline drifts.
Temporal Augmentations: Random temporal cutoffs, time warping, and SpecAugment-style time masking encourage invariance to movement latency and speed variations.
Channel Augmentation: Channel dropout simulates electrode failures.
Gaussian Smoothing: Applied in both training and validation to smooth binned spike rates.

The training objective combines CTC loss with an entropy regularization term to prevent the model from collapsing into over-peaked blank predictions.

Key Results
Evaluated on the held-out validation set (1,426 sentences), the proposed system achieves a Character Error Rate (CER) of 23.80%.

Comparison: This represents a relative improvement of 20.67% over the baseline GRU + CTC system (30.00% CER) provided with the benchmark.
Ablation: Removing the session-specific alignment layer increased CER to 24.28%, while removing data augmentation caused a significant performance drop to 39.55%, highlighting the critical role of augmentation in generalization.
Error Analysis: The analysis reveals that performance variability is primarily driven by inter-session signal degradation, with later sessions showing higher error rates (up to 40–45%) compared to earlier sessions (~10%). The dominant error mode is incorrect word boundary segmentation (e.g., space deletion or insertion within words) rather than phonetic confusion, accounting for the majority of top substitution errors.

Significance and Claims
The paper claims to demonstrate that effective character-level decoding is achievable in a fully end-to-end framework without relying on external language model rescoring. By eliminating the need for large, memory-intensive LMs and sequential beam search, the proposed approach offers a pathway toward simpler, more deployable, and real-time intracortical speech neuroprostheses. The work provides a strong neural signal for downstream linguistic processing, suggesting that the neural decoder itself can learn sufficient structure to produce meaningful text, even if word-level accuracy currently lags behind systems utilizing external LMs. The authors position this as a step toward decoupling neural signal extraction from downstream linguistic processing, addressing the latency and hardware constraints of fully implantable devices.