Throat and acoustic paired speech dataset for deep learning-based speech enhancement

1. Problem Statement

Capturing clear speech in high-noise environments (e.g., factories, subways) using standard acoustic microphones is difficult due to background noise. Body-Conducted Microphones (BCMs), specifically throat microphones, offer a solution by capturing vibrations directly from the skin near the vocal cords, inherently suppressing airborne noise. However, BCMs suffer from significant limitations:

• Low-Pass Effect: Sound transmission through skin and tissue attenuates high-frequency components, degrading speech clarity.
• Loss of Critical Phonemes: Voiceless obstruents (e.g., /s/, /z/, /h/) and plosives are poorly captured because they rely on air turbulence rather than vocal cord vibration.
• Lack of Standardized Data: Progress in deep learning-based speech enhancement for BCMs is hindered by the absence of large-scale, standardized, and meticulously aligned datasets. Existing datasets are often small, proprietary, or lack precise synchronization between BCM and acoustic signals.

2. Methodology

A. Dataset Construction (TAPS)

The authors introduced the Throat and Acoustic Paired Speech (TAPS) dataset, designed specifically for training Throat Microphone Speech Enhancement (TMSE) models.

• Participants: 60 native Korean speakers (balanced gender, mean age ~27).
• Hardware Setup:
  • Throat Microphone: A custom MEMS accelerometer (TDK IIM-42652) mounted on the supraglottic area (above the thyroid cartilage) to capture skin vibrations. Sampled at 8 kHz.
  • Acoustic Microphone: A MEMS microphone (CUI Devices) placed 30 cm from the lips. Sampled at 16 kHz.
  • Environment: An acoustically treated room with high isolation and low reverberation.
• Content: 6,000 unique sentences from a Korean newspaper corpus, distributed as 100 sentences per speaker.
• Data Splits:
  • Train: 40 speakers (4,000 utterances, 10.2 hours).
  • Dev: 10 speakers (1,000 utterances, 2.5 hours).
  • Test: 10 speakers (1,000 utterances, 2.6 hours).

B. Signal Processing & Alignment

A critical contribution of the methodology is the handling of temporal misalignment between the two modalities.

• Sources of Mismatch: The authors identified three factors causing timing differences: (1) distance between lips and acoustic mic, (2) individual vocal tract anatomy, and (3) phoneme-specific resonance locations.
• Alignment Strategy: Instead of a simple global shift, the authors analyzed mismatch variations across speakers and sentences. They compared three correction strategies:
  1. Per-utterance correction.
  2. Per-speaker mean correction.
  3. Overall mean mismatch correction (Global).
  • Result: The Overall Mean Mismatch Correction (applying a global shift of 13 samples at 8 kHz, or ~1.625 ms) yielded the most consistent improvements in model performance across different metrics.
• Preprocessing: Included gravitational offset removal (high-pass filtering), noise reduction on acoustic targets using Demucs, trimming, and upsampling the throat signal to 16 kHz.

C. Baseline Models

Three deep learning architectures were trained on the TAPS dataset to evaluate the enhancement task:

1. TSTNN (Two-Stage Transformer Neural Network): A masking-based approach.
2. Demucs: A mapping-based approach using a convolutional encoder-decoder with BLSTM.
3. SE-conformer: A mapping-based approach combining encoder-decoder structures with Conformer blocks.

3. Key Contributions

1. The TAPS Dataset: The first publicly available, large-scale, standardized dataset of paired throat and acoustic speech recordings from 60 speakers, specifically positioned on the supraglottic area for optimal signal-to-noise ratio.
2. Optimized Alignment Pipeline: A systematic analysis of signal mismatch factors and the validation of an "Overall Mean Mismatch Correction" strategy, which is critical for training robust deep learning models on multi-modal sensor data.
3. Benchmarking TMSE: A comprehensive evaluation of state-of-the-art models (TSTNN, Demucs, SE-conformer) on the task of reconstructing high-frequency and voiceless components from throat signals.
4. Open Source Resources: Public release of the dataset (Hugging Face, Zenodo), hardware design files, firmware, and baseline training code.

4. Results

The study evaluated models using objective speech quality metrics (PESQ, STOI, CSIG, CBAK, COVL) and speech content restoration metrics (CER, WER via Whisper ASR).

• Performance Comparison:
  • Mapping-based models (Demucs, SE-conformer) significantly outperformed the masking-based model (TSTNN) in content restoration.
  • SE-conformer achieved the best overall performance:
    • PESQ: 1.971 (vs. 1.22 for raw throat mic).
    • STOI: 0.892.
    • CER: 24.4% (vs. 84.4% for raw throat mic).
• Key Finding: Mapping-based approaches are superior for TMSE because they can generate missing high-frequency and voiceless components, whereas masking-based approaches can only modify existing spectral components and fail to recover lost information.
• Impact of Alignment: Using the optimized global alignment strategy improved PESQ and STOI scores and significantly reduced Character Error Rates (CER) compared to uncorrected data.

5. Significance

• Advancing Wearable Tech: The TAPS dataset and findings provide a foundation for developing robust speech enhancement systems for wearable devices, enabling clear communication in extreme noise environments (industrial, military, public transit).
• Standardization: By establishing a standardized dataset and alignment protocol, the paper facilitates reproducibility and direct quantitative comparison across future research in body-conducted speech.
• Broader Applications: The technology supports advancements in Automatic Speech Recognition (ASR) for noisy environments, silent speech interfaces, and assistive communication technologies.
• Methodological Extension: The framework for dataset construction and alignment can be extended to other languages and diverse domains such as human-computer interaction.