⚡ electrical engineering

emg2speech: Synthesizing speech from electromyography using self-supervised speech models

This paper presents a neuromuscular speech interface that leverages the strong linear relationship between self-supervised speech representations and electromyographic (EMG) signals to synthesize audible speech directly from silent articulation, successfully demonstrating the system on a participant with amyotrophic lateral sclerosis (ALS).

Original authors: Harshavardhana T. Gowda, Daniel C. Comstock, Lee M. Miller

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Harshavardhana T. Gowda, Daniel C. Comstock, Lee M. Miller

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a superpower: the ability to speak without opening your mouth. You just think the words, your tongue and lips move slightly inside your mouth, and a machine instantly turns those tiny, silent movements into a clear, audible voice.

This is the dream behind a new technology called EMG-to-Speech, and a team of researchers from UC Davis has just built a major step toward making it a reality. Here is how they did it, explained simply.

The Problem: The "Silent" Speaker

For people with conditions like ALS (Lou Gehrig's disease) or those who have had their voice box removed, speaking is impossible. They can still move their lips and tongue, but no sound comes out.

Previous attempts to help them used invasive brain implants (surgery inside the skull) or tried to guess what they were saying based on messy muscle signals. These methods were often expensive, risky, or required the user to speak out loud while recording, which doesn't help people who are completely silent.

The Solution: The "Muscle-to-Music" Translator

The researchers created a system that listens to the electrical signals of the muscles in your face and neck (called EMG) and translates them directly into speech.

Here is the clever trick they used, explained with an analogy:

1. The "Secret Code" (Self-Supervised Models)

Imagine you have a giant library of books (a massive AI model trained on thousands of hours of human speech). This library knows the "secret code" of how humans make sounds. It knows that to make an "O" sound, your lips must round, and to make an "S," your teeth must touch.

The researchers realized that this AI library doesn't just know sound; it implicitly knows movement. It understands the physical mechanics of speech.

2. The "Muscle Fingerprint"

When you try to say a word silently, your facial muscles twitch. The researchers found that these tiny electrical twitches create a specific pattern, like a fingerprint.

The Discovery: They found a direct, simple mathematical link between the "secret code" in the AI library and the "muscle fingerprint."
The Analogy: Think of the AI library as a master chef who knows exactly how to make a cake. The muscle signals are the ingredients sitting on the counter. The researchers realized that if you look at the ingredients (muscle power), you can predict exactly what the chef (the AI) is thinking about making, even if you can't hear the oven yet.

3. The "Bridge" (The New System)

Instead of trying to build a complex machine to guess the sound from the muscle (which is like trying to guess a song just by looking at a dancer's shoes), they built a bridge.

They take the muscle signals.
They translate them into the AI's "secret code" (the representation of speech).
Then, they use a pre-made "voice synthesizer" (a tool that turns text/code into audio) to play the sound.

This is like taking a silent movie, translating the actors' gestures into a script, and then having a narrator read the script aloud.

Why This is a Big Deal

1. It Works for "Silent" Speakers
Most previous systems needed you to speak out loud while recording to teach the computer. This system works even if you are completely silent. They tested it on a woman with ALS who could not make a single sound, and the system successfully turned her silent lip movements into audio.

2. It's Non-Invasive
You don't need brain surgery. They just put sticky electrodes (like those used in gym physiotherapy) on the skin of the neck and face. It's safe, cheap, and easy to wear.

3. It's "Open Source"
The researchers didn't just keep the secret; they released the "recipe" (the code) and the "ingredients" (the data) to the public. They recorded hours of data from a healthy person and the person with ALS, so other scientists can build on this work immediately.

The Results: How Good Is It?

The system isn't perfect yet (it's like a new car that runs but needs a few tune-ups).

Accuracy: When the system spoke for the healthy person, it got the words right about 62% of the time. For the person with ALS, it was about 50%.
The "Human" Test: When humans listened to the generated voice, they rated it as "understandable" and "natural-sounding," even if they couldn't catch every single word.

The Future

Think of this as the "iPhone 1" of silent speech. It proves the concept works without surgery. The researchers are now working on making it faster, more accurate, and able to handle the daily changes in how our skin and muscles behave (like when you're tired or the electrodes shift slightly).

In short: They found a way to read the "silent movie" of your facial muscles and project it as a loud, clear voice, giving a voice back to those who have lost theirs, all without a single drop of blood or a single surgery.

1. Problem Statement

The paper addresses the challenge of creating a non-invasive speech interface for individuals who have lost the ability to speak due to conditions like Amyotrophic Lateral Sclerosis (ALS), laryngectomy, or dysarthria.

The Core Challenge: Converting electromyographic (EMG) signals recorded from orofacial muscles directly into intelligible audio.
Key Difficulty: Unlike previous works that assume known temporal alignment between input signals (EMG) and output audio, this work tackles the alignment-free setting. In clinical scenarios (e.g., silent articulation by ALS patients), parallel EMG-audio pairs are often unavailable or unreliable. The model must learn the mapping and infer the underlying temporal alignment simultaneously from an exponential search space.
Limitations of Prior Work: Existing methods often rely on small vocabularies, require invasive neural implants (BCIs), or depend on time-aligned training data, making them difficult to reproduce or deploy clinically.

2. Methodology

The authors propose a pipeline that maps EMG signals to self-supervised speech (S3) representations, which are then synthesized into audio using a pre-trained vocoder.

A. Data Collection

The study introduces three datasets:

DATA GENERAL: ~9 hours of EMG data from a healthy participant articulating ~6,800 unique words (large vocabulary).
DATA ALS: ~~1 hour of silent articulation EMG data from an ALS patient (~~300 unique words).
DATA OROFACIAL GESTURES: Discrete muscle movements from 12 healthy participants to analyze articulatory structure.

Sensors: 31 monopolar surface electrodes placed on the neck, chin, jaw, cheeks, and lips, sampled at 5000 Hz.

B. Feature Representation & Theoretical Insight

EMG Features: The authors analyze EMG covariance matrices ( $E$ ) and their diagonals ( $D(E)$ ), which represent muscle action potential power. They find that $D(E)$ vectors form distinct, separable clusters for different articulatory gestures.
Self-Supervised Speech (S3) Representations: They utilize pre-trained models (HuBERT, Wav2Vec 2.0, WavLM) to extract high-dimensional speech features ( $H$ ).
Key Discovery: A strong linear relationship exists between S3 representations ( $H$ ) and EMG power ( $D(E)$ ). A simple linear mapping predicts EMG power from S3 features with a correlation of $r = 0.85$ . This suggests S3 models implicitly encode articulatory mechanisms reflected in muscle activity.

C. Model Architecture

The system uses an alignment-free sequence-to-sequence approach:

Encoder: A Time-Depth Separable (TDS) convolutional network processes EMG inputs (vectorized covariance $vec(E)$, power $D(E)$ , or spectrograms $vec(B)$). The architecture includes a "shift-tolerant" module to handle spatial variability in electrode placement.
Target Space: Instead of predicting raw audio or phonemes directly, the encoder predicts discrete HuBERT units ( $dis(H)_{HuBERT}$ ) derived from text transcripts (using Google TTS for training targets).
Training Objective: The model is trained using Connectionist Temporal Classification (CTC) loss, which allows learning without frame-level EMG-audio alignment.
Phoneme-Guided Decoding: To improve performance, a multi-task learning approach is used. The model predicts both HuBERT units and phonemes simultaneously. A consistency loss ensures the predicted HuBERT units align with the predicted phoneme distribution, leveraging the fact that phonemes are defined by articulatory configurations captured by EMG.
Synthesis: The predicted discrete HuBERT units are fed into a pre-trained Tacotron vocoder to generate the final audio waveform.

3. Key Contributions

New Datasets: Open-sourcing one of the largest high-quality EMG-to-speech datasets to date, including a large-vocabulary corpus (healthy) and a small-vocabulary corpus (ALS), addressing the lack of reproducible benchmarks in this field.
Theoretical Link: Establishing the first quantitative link showing that S3 representations linearly map to EMG power, proving that S3 models implicitly encode articulatory structure.
Alignment-Free Architecture: Designing a lightweight encoder that operates effectively without time-aligned EMG-audio pairs, making it suitable for clinical populations where silent articulation is the only option.
Phoneme-Guided Decoding: Introducing a method to use phonetic structure to guide the decoding of discrete speech units, significantly improving synthesis quality.

4. Results

Linear Mapping: The correlation between S3 features and EMG power reached $r = 0.85$ , with HuBERT Layer 6 showing the strongest predictive power.
Unit Error Rate (UER): On the healthy participant dataset (DATA GENERAL), using full covariance EMG features ($vec(E)$) with phoneme-guided decoding achieved a UER of 51.81% (down from 56.08% with unit-only training).
Word Error Rate (WER):
- Healthy Participant: Average WER of 62.55% (human transcription).
- ALS Participant: Average WER of 50.82% (human transcription) on silent articulation.
Comparison: While WER is higher than invasive BCIs (which often report ~40-50%), this is achieved non-invasively and without time-aligned training data. The system outperforms previous non-invasive EMG benchmarks that relied on smaller vocabularies or aligned data.
Mean Opinion Score (MOS): The synthesized audio received a MOS of 3.79/5.0, indicating perceptual plausibility despite high WER, as the acoustic waveform remains speech-like.

5. Significance and Impact

Clinical Applicability: The system demonstrates a viable path for restoring functional speech to patients with severe paralysis or laryngectomy without requiring surgical brain implants.
Reproducibility: By releasing large datasets and code, the authors address the "black box" nature of prior EMG-to-speech research, enabling fair comparison and future advancement.
Efficiency: The approach works well with limited data (as little as 40 minutes for the ALS participant), which is crucial given the difficulty of collecting large-scale EMG datasets from clinical populations.
Future Direction: The work highlights that leveraging the geometric structure of EMG signals and the articulatory knowledge embedded in self-supervised speech models is a promising direction for non-invasive neuroprosthetics.

In summary, emg2speech bridges the gap between muscle activity and speech synthesis by leveraging the latent articulatory structure found in self-supervised speech models, offering a non-invasive, reproducible, and effective solution for silent speech communication.