An Approach to Simultaneous Acquisition of Real-Time MRI Video, EEG, and Surface EMG for Articulatory, Brain, and Muscle Activity During Speech Production

This paper presents a novel framework for the simultaneous acquisition of real-time MRI, EEG, and surface EMG to capture brain, muscle, and articulatory activity during speech, featuring a specialized artifact suppression pipeline to overcome technical challenges and enable unprecedented insights into speech neuroscience.

The Gist

Imagine trying to understand how a symphony orchestra plays a piece of music. If you only listen to the final sound coming out of the concert hall, you get the melody, but you miss the conductor's cues, the violinist's finger movements, and the drummer's heartbeat. You don't know how the music was made, only what was made.

For decades, scientists studying human speech have been stuck in a similar position. They could hear the voice (the audio) or see the mouth moving (via cameras), but they couldn't easily watch the brain's "conductor" and the muscles' "finger movements" happening at the exact same time.

This paper introduces a groundbreaking new "super-camera" setup that finally lets us watch the entire speech orchestra in real-time. Here is the simple breakdown:

The Three Musicians in the Orchestra

The researchers built a system to record three different things simultaneously while a person speaks:

1. The Brain (EEG): Like a high-speed camera capturing the conductor's hand signals. It shows the electrical sparks in the brain that decide what to say.
2. The Muscles (EMG): Like sensors on the violinist's fingers. It tracks the tiny electrical signals in the face and throat muscles that prepare to move.
3. The Mouth (Real-Time MRI): Like a super-fast X-ray movie. It shows the tongue, lips, and jaw physically moving to shape the sound.

The Big Challenge: The "Noisy Room"

Trying to do this is incredibly difficult because of the environment. To get the MRI video, the person has to lie inside a giant, powerful magnet (the MRI machine).

• The Problem: MRI machines are like giant, noisy radio stations. They blast out electromagnetic waves that scramble the delicate brain and muscle signals, turning them into static noise. It's like trying to hear a whisper while standing next to a jet engine.
• The Solution: The team invented a special "noise-canceling" pipeline. Think of it as a sophisticated audio editor that knows exactly what the "jet engine" noise looks like. It creates a template of the noise and subtracts it from the recording, leaving only the clear whisper of the brain and muscles.

What They Discovered

Using this new setup, they tested it on a person speaking, whispering, and even imagining speaking (saying words in their head without moving their lips).

1. The "Ghost" Movements: Even when the person tried to speak silently in their head, the MRI camera caught tiny, almost invisible movements in the mouth (like a twitch of the tongue). It's as if the body is practicing the dance steps even when the dancer is sitting still.
2. The Brain's Map: After cleaning up the noise, they could clearly see the brain lighting up in the left side (where language lives), proving that even silent speech activates the same language centers as loud speech.
3. No Interference: They proved that wearing the brain sensors and muscle wires didn't ruin the MRI movie. The "costume" didn't distract the "camera."

Why This Matters (The Future)

This isn't just about watching people talk; it's about building the future of communication technology.

• For Brain-Computer Interfaces (BCI): Imagine a person who has lost their ability to speak due to an injury. Currently, computers struggle to guess what they want to say just by reading brain waves. With this new method, scientists can finally train computers to understand the link between "Brain Signal" and "Mouth Movement" directly. This could lead to devices that let paralyzed people "speak" by thinking, with much higher accuracy.
• For Understanding Speech Disorders: It gives doctors a complete map of what goes wrong in conditions like stuttering or apraxia, showing exactly where the "conductor" and the "musicians" are out of sync.

In a Nutshell

This paper is the first time scientists successfully hooked up a brain monitor, a muscle tracker, and a moving X-ray camera all at once. They figured out how to filter out the loud noise of the MRI machine to see the quiet signals of speech. It's like finally getting a clear, high-definition view of the entire speech production chain, opening the door to better treatments for speech disorders and revolutionary new ways for humans to communicate with machines.

Technical Summary

Here is a detailed technical summary of the paper "An Approach to Simultaneous Acquisition of Real-Time MRI Video, EEG, and Surface EMG for Articulatory, Brain, and Muscle Activity During Speech Production."

1. Problem Statement

Speech production is a complex physiological chain involving neural planning, motor control, muscle activation, and articulatory kinematics. While acoustic signals are the most accessible output, they do not reveal the underlying causal neurophysiological mechanisms.

• The Gap: Previous studies have attempted to combine subsets of these modalities (e.g., EEG + EMA, or MEG + articulography), but none have successfully captured the full chain simultaneously: brain activity (EEG), muscle activation (surface EMG), and real-time articulatory movement (rtMRI).
• Technical Challenges: Simultaneous acquisition of these three modalities inside an MRI scanner faces severe signal contamination issues:
  1. Gradient Artifacts (GA): Rapid magnetic field switching induces large voltage transients in EEG/EMG leads.
  2. Ballistocardiogram (BCG) Artifacts: Cardiac-driven motion within the static magnetic field creates quasi-periodic noise.
  3. Myogenic Contamination: Speech production causes facial muscle activity and head/jaw motion, creating artifacts that overlap with neural signals and perturb electrode contacts.

2. Methodology

Experimental Setup

• Hardware:
  • MRI: 0.55T scanner using a custom 8-channel upper airway coil with a spiral bSSFP sequence (TR = 5.05 ms, 99 fps) for real-time vocal tract imaging.
  • EEG/EMG: MR-compatible BrainVision system (16 electrodes: 9 EEG, 2 EOG, 3 EMG, 1 ECG).
  • Synchronization: Fiber-optic triggers synchronize the MRI clock with the electrophysiology system (sampled at 5 kHz).
  • Audio: Optical microphone for concurrent audio recording.
• Subject & Tasks: A pilot study with one male native English speaker. Tasks included fully phonated speech, silent speech, and imagined speech. The study focused primarily on phonated speech using 18 disyllabic nonce words (VCV structure).
• Stimulus Protocol: Randomized inter-trial intervals with fixation crosses and "Go" signals to prevent time-locking of neural responses to the MRI acquisition cycle.

Data Processing Pipeline (Artifact Suppression)

The authors developed a multi-stage denoising pipeline tailored for this tri-modal setting:

1. Gradient Artifact Correction: Uses Average Artifact Subtraction (AAS). A local template is constructed by averaging sliding windows of gradient repetitions and subtracted from the data to remove periodic transients.
2. Pulse (BCG) Artifact Correction: Uses ECG-informed AAS. R-peaks are detected from the ECG channel; cardiac-locked segments are averaged within a sliding window and subtracted to remove pulse-related deflections.
3. Myogenic & Ocular Artifact Correction: Uses Reference-based Canonical Correlation Analysis (CCA).
   • EEG channels are jointly decomposed with reference signals (3 EMG channels for muscle, 2 EOG channels for eye movements).
   • Components with high correlation ($\rho > 0.4$) to the reference signals are identified as artifacts and removed via orthogonal projection.
   • This step targets non-neural components (blinks, jaw motion) while preserving cortical activity.

3. Key Contributions

• First-of-its-Kind Acquisition: This is the first study to simultaneously record rtMRI, EEG, and surface EMG during speech production, providing a complete view of the speech production chain.
• Novel Artifact Suppression Framework: Introduction of a specialized pipeline combining AAS (for MRI artifacts) and CCA (for myogenic/ocular artifacts) specifically adapted for the tri-modal environment.
• Validation of Compatibility: Demonstration that the EEG/EMG hardware does not significantly degrade rtMRI image quality (SNR analysis showed no significant impact on articulatory regions).
• Imagined Speech Observation: Successful detection of micro-articulatory movements (e.g., velum movement) during "imagined" speech tasks where subjects were instructed to remain silent, challenging the assumption of complete motor inhibition.

4. Results

• Temporal Alignment: The MRI and EEG systems were intrinsically aligned with an average duration difference of only 8.3 ms/s, well within the rtMRI frame size (10.1 ms).
• Artifact Removal Efficacy:
  • Gradient/BCG: Raw EEG inside the scanner showed large-amplitude periodic transients. Post-correction, these were substantially suppressed, yielding temporal and spectral characteristics comparable to recordings taken outside the scanner.
  • Correlation: The corrected "inside-scanner" EEG showed an average temporal correlation of 0.66 with the "outside-scanner" reference, rising to ~0.8 in frontal/language areas (FPz, C3, F3).
  • Myogenic/ocular: Before correction, frontal artifacts reached ~60 µV. After CCA denoising, amplitudes dropped to ~20 µV, revealing left-lateralized activation consistent with known language processing networks.
• Micro-Movements: rtMRI captured subtle articulatory movements during imagined speech, providing an objective measure of motor execution even without acoustic output.

5. Significance and Future Applications

• Brain-Computer Interfaces (BCI):
  • Provides a "physiological ground truth" for silent and imagined speech.
  • Enables the training of robust decoders that map neural/muscle activity directly to articulatory trajectories, bypassing the need for acoustic signals.
  • Helps distinguish between intended speech and involuntary micro-movements that may degrade BCI performance.
• Speech Science & Neuroscience:
  • Offers a comprehensive window into the spatiotemporal dynamics of speech, from planning to execution.
  • Aids in distinguishing feedforward motor commands from sensory feedback loops.
  • Potential to identify neural breakdowns in speech disorders (e.g., stuttering, apraxia).
• Limitations & Future Work:
  • Current limitations include a single-subject pilot study, passive EEG electrodes (higher noise than active), and limited EMG montage.
  • Future work aims to scale to larger cohorts, use high-density EEG, and investigate more linguistically complex tasks.

Conclusion: This study establishes a technically feasible framework for simultaneous tri-modal acquisition, bridging the gap between neural intent, motor execution, and physical articulation. It lays the groundwork for next-generation BCIs and deeper insights into the neurophysiology of human speech.