mmWave Radar Aware Dual-Conditioned GAN for Speech Reconstruction of Signals With Low SNR

This paper proposes RAD-GAN, a two-stage radar-aware generative adversarial network with a Multi-Mel Discriminator and Residual Fusion Gate, which effectively reconstructs intelligible full-bandwidth speech from noisy, low-SNR mmWave signals captured through glass walls, outperforming state-of-the-art methods even with limited data and no pre-trained modules.

The Gist

Imagine you are trying to listen to a friend whispering a secret to you, but there is a thick glass wall between you, and a loud construction crew is working right next to you. The sound that reaches your ear is a muddy, garbled mess of static and faint vibrations. Now, imagine you don't have ears, but a special "super-vision" radar that can see the tiny vibrations of your friend's throat through that glass.

That is the challenge this paper tackles: How do we turn a muddy, noisy radar vibration into clear, understandable speech?

Here is the story of how the researchers solved this, explained in simple terms.

The Problem: The "Muddy Radio"

Millimeter-wave (mmWave) radar is like a super-sensitive camera that sees vibrations instead of light. It's great because it can "hear" through walls and doesn't need a microphone in the room. But there's a catch:

1. It's Band-Limited: It's like trying to listen to a radio station that only plays the bass notes. The high-pitched sounds (like "s," "t," and "f" sounds) are missing.
2. It's Noisy: The signal is buried under static, like trying to hear a whisper in a hurricane.
3. The Data is Scarce: They didn't have millions of examples to learn from; they had a relatively small dataset.

The Solution: The "RAD-GAN" Chef

The team built a two-step cooking recipe (a pipeline) to turn this muddy radar soup into a delicious, clear meal (speech). They call their system RAD-GAN.

Think of it like a master chef (the Generator) who needs to recreate a complex dish based on a very blurry, low-quality photo of it.

Step 1: The "Blindfolded Practice" (Pre-training)

Before the chef tries to cook the real, messy meal, they practice in a controlled kitchen.

• The Trick: They take clear, perfect audio and artificially chop off the high notes (making it sound like the radar).
• The Goal: They teach the chef to guess what the missing high notes should sound like based on the low notes.
• Why? This teaches the system the basic rules of how speech works without getting confused by the real-world noise yet. It's like learning to draw a perfect circle before trying to draw a portrait in a shaking car.

Step 2: The "Master Chef's Assistants" (Fine-tuning)

Now, the chef faces the real challenge: the noisy radar data. But they don't go in alone. They have two special assistants:

1. The "Radar Translator" (WaveVoiceNet): This is a helper that looks at the noisy radar vibration and tries to make a "best guess" at what the speech sounds like. It's not perfect, but it's a good starting point.
2. The "Smart Mixer" (Residual Fusion Gate): This is the most clever part. Imagine the Radar Translator gives you a sketch, and the original noisy radar gives you a blurry photo. The Smart Mixer looks at both and says, "Okay, I'll trust the sketch for the big shapes, but I'll use the blurry photo for the fine details where the sketch is wrong." It blends the two inputs perfectly, keeping the good parts and ignoring the bad.

The "Taste Testers" (The Discriminators)

To make sure the chef isn't just making up nonsense, they have three strict food critics (Discriminators):

• The Rhythm Critic: Checks if the speech sounds natural and rhythmic.
• The Detail Critic: Checks if the tiny sounds are crisp.
• The "Mel" Critic (New!): This is a special critic that looks at the spectrum of the sound (like a visual map of frequencies). It ensures the "flavor" of the sound matches real human speech, not just a robotic beep.

Why This is a Big Deal

Most other systems try to solve this by eating a massive amount of data (like reading a million cookbooks) or using pre-trained models that are already "smart."

This team did something different:

• They learned with less: They didn't need a massive dataset.
• They didn't cheat: They didn't use pre-trained models that already knew the answer.
• They worked in the dark: They succeeded even when the signal was extremely noisy (so quiet it was almost silent).

The Result

When they tested their system, it didn't just "okay." It was the best at the task.

• It reconstructed speech that sounded more natural to human ears.
• It preserved the "silence" parts of the conversation better (so it didn't sound like static when the person stopped talking).
• It captured the sharp edges of words (like "stop" or "cat") much better than previous methods.

The Bottom Line

The researchers built a smart, two-step system that acts like a noise-canceling, imagination-powered translator. It takes a faint, broken vibration from a radar, uses a "best guess" helper, and a "smart mixer" to fill in the missing pieces, resulting in clear, intelligible speech even when the original signal was barely there.

It's like taking a blurry, black-and-white photo of a face and using AI to not only sharpen the image but also guess the missing colors and details so perfectly that it looks like a high-definition color photo.

Technical Summary

1. Problem Statement

The paper addresses the challenge of reconstructing intelligible, full-bandwidth speech from Millimeter-wave (mmWave) radar signals.

• Core Difficulty: mmWave radar captures are inherently band-limited (typically 0–1 kHz) and suffer from extremely low Signal-to-Noise Ratios (SNR), ranging from -5 dB to -1 dB.
• Environmental Constraints: The signals are often captured through obstructions (e.g., glass walls) or via secondary surface vibrations (e.g., aluminum foil), leading to severe noise contamination and phase degradation.
• Limitations of Existing Work: Current state-of-the-art approaches often rely on large-scale datasets, pre-trained models, or extensive compute resources, and they frequently fail to perform well under realistic, low-SNR deployment scenarios without data augmentation.

2. Methodology: RAD-GAN

The authors propose a Radar-Aware Dual-Conditioned Generative Adversarial Network (RAD-GAN) operating via a two-stage training pipeline. The system is designed to perform bandwidth extension (from 1 kHz to 4 kHz) without relying on external pre-trained modules or data augmentation.

A. Architecture Components

1. Generator (HiFi-GAN based):
   • Uses the standard HiFi-GAN generator architecture (transposed-convolution upsampling with Multi-Receptive Field blocks).
   • Conditioning: Unlike standard HiFi-GAN which uses only mel-spectrograms, this generator is conditioned on fused mel-spectrograms derived from two sources: the noisy radar input and an enhanced output from a WaveVoiceNet module.
2. Discriminators:
   • Standard Discriminators: Multi-Period Discriminator (MPD) and Multi-Scale Discriminator (MSD) for waveform-level supervision.
   • Proposed Multi-Mel Discriminator (MMD): A novel mmWave-tailored discriminator. It consists of two parallel 2D convolutional branches operating on mel-spectrograms:
     • One branch uses Spectral Normalization (for stability).
     • The other uses Weight Normalization (for flexibility).
     • This provides complementary time-frequency signals to improve spectral realism and training stability, which is crucial when raw waveform supervision is unreliable due to noise.
3. WaveVoiceNet (WVN) Module:
   • Used as an auxiliary conditioning branch. It is strong at magnitude-domain transformation but less reliable for phase. It provides an "enhanced" mel-spectrogram to guide the generator.
4. Residual Fusion Gate (RFG):
   • A novel mechanism to fuse the noisy radar mel-spectrogram ($M_n$) and the WVN-enhanced mel-spectrogram ($M_w$).
   • It learns a local mask ($G$) and a global scale to decide how much to trust the WVN residual correction ($M_w - M_n$) versus the baseline noisy input.
   • Logic: If WVN cues are unreliable, the gate falls back to the noisy baseline; if reliable, it amplifies the WVN correction.

B. Two-Stage Training Strategy

1. Phase 1: Pre-training (Spectral Reconstruction)
   • Goal: Learn stable bandwidth extension without GAN instability.
   • Data: Synthetically clipped clean speech (band-limited to 1 kHz).
   • Loss: Only spectral reconstruction losses (L1 Mel loss with high-frequency weighting + Multi-Resolution STFT loss). No adversarial losses are used.
   • Outcome: The generator learns a reliable mapping from low-frequency to high-frequency content.
2. Phase 2: Fine-tuning (Adversarial Refinement)
   • Goal: Adapt to real, noisy radar data and improve perceptual quality.
   • Data: Real mmWave radar recordings (Task 1: direct vibration; Task 2: secondary surface vibration).
   • Process: The generator is initialized from Phase 1. The RFG fuses noisy inputs with WVN outputs to create the conditioning input.
   • Loss: Combines pre-training reconstruction losses with adversarial losses (from MPD, MSD, and the new MMD) and feature-matching losses.

3. Key Contributions

1. RAD-GAN Architecture: A novel pipeline for mmWave-to-speech reconstruction that achieves bandwidth extension on signals with SNRs as low as -5 dB.
2. Multi-Mel Discriminator (MMD): A specialized 2D mel-spectrogram discriminator with dual normalization branches to ensure stable training in noisy, low-data regimes.
3. Residual Fusion Gate (RFG): A mechanism to dynamically fuse noisy and enhanced conditioning channels, allowing the model to adaptively rely on the most reliable signal source.
4. Two-Stage Training Strategy: A pre-training + fine-tuning approach that isolates bandwidth extension learning from GAN instability, enabling high performance on limited datasets without data augmentation.

4. Experimental Results

The model was evaluated on the RASE 2026 Challenge dataset (TI AWR2243BOOST radar), consisting of two tasks:

• Task 1: Direct diaphragm vibration.
• Task 2: Secondary surface vibration (Aluminum foil), representing a more challenging, noisier scenario.

Performance Metrics:

• Metrics Used: PESQ (Perceptual Evaluation of Speech Quality), ESTOI (Extended Short-Time Objective Intelligibility), MFCC Cosine Similarity (CS), and DNSMOS (Deep Noise Suppression Mean Opinion Score).
• Comparison: RAD-GAN (M6) outperformed baselines including WaveVoiceNet (M0), HiFi-GAN (M1), DCCTN (M2), and various diffusion-based models (M4, M5).
• Key Findings:
  • Weighted Score: RAD-GAN achieved the highest overall weighted score (0.333), significantly beating WaveVoiceNet (0.260) and HiFi-GAN (0.288).
  • Task 2 Performance: It showed particular strength in the difficult Task 2 (0.297 score), demonstrating robustness to extreme noise.
  • Qualitative Analysis: Waveform and spectrogram visualizations showed RAD-GAN produced clearer upper-band harmonics, better preserved silence regions (less leakage), and sharper onsets/offsets compared to baselines.
  • Ablation Study: The study confirmed that while adding MMD+MR-STFT provided a small gain, the pre-training stage (+0.022) and WVN conditioning (+0.021) were the primary drivers of performance improvement.

5. Significance

• Practical Deployment: The method works effectively with limited data (approx. 42 hours of paired speech) and no data augmentation, making it suitable for real-world scenarios where collecting massive datasets is impractical.
• Robustness: It addresses the critical gap of reconstructing speech through obstructions (glass/foil) and in low-SNR environments where conventional microphones fail.
• Efficiency: By avoiding pre-trained large language models or massive diffusion backbones, the solution is computationally efficient and tailored specifically for the physics of mmWave sensing.
• Future Impact: The authors plan to focus on real-time deployment (latency reporting) and model compression for edge inference, paving the way for privacy-preserving, contact-free speech monitoring in complex environments.