NasoVoce: A Nose-Mounted Low-Audibility Speech Interface for Always-Available Speech Interaction

NasoVoce is a nose-mounted speech interface that fuses acoustic and vibration signals to enable robust, discreet, and always-available voice interaction for AI, effectively overcoming the limitations of existing silent and whispered speech recognition methods.

The Gist

Imagine you are walking down a busy street, sitting in a crowded café, or riding a noisy train. You want to talk to your AI assistant to get directions, check your schedule, or brainstorm an idea. But you can't shout out loud; it's rude, it might leak your private thoughts, and the background noise would drown you out.

This is the problem NasoVoce solves. It's a new way to talk to computers that is quiet, private, and works even when the world is loud.

Here is how it works, explained simply:

The Problem with Current Tech

Think of trying to have a conversation in a hurricane.

• Normal Microphones (like on your phone): These are like sensitive ears. They hear everything, but if a truck drives by, they can't tell the difference between your voice and the truck.
• Bone Conduction (like some headphones): These are like feeling the vibrations in your throat. They are great at ignoring outside noise, but they struggle to hear "whispers" because whispers don't vibrate your throat much. They are also often bulky or uncomfortable.
• Lip Reading Cameras: These are like a security camera on your face. They work well, but they are creepy, require good lighting, and you have to look at a screen.

The NasoVoce Solution: The "Super-Sensor" on Your Nose

The researchers built a tiny device that fits right on the nose pads of smart glasses (the part that rests on your nose). It has two special "ears":

1. The Air Ear (Microphone): This listens to the air. It hears your voice clearly when it's quiet, but it gets confused by loud noises.
2. The Bone Ear (Vibration Sensor): This feels the vibrations traveling through your nose and skin. It ignores the loud truck outside because the truck doesn't vibrate your nose, but it's a bit "muffled" like hearing someone talk through a pillow.

The Magic Trick:
The genius of NasoVoce is that it fuses these two ears together using a smart computer brain (AI).

• When it's quiet, the AI trusts the "Air Ear."
• When it's loud, the AI leans heavily on the "Bone Ear" to filter out the noise.
• The Result: It creates a "Super Voice" that is as clear as a normal voice but as quiet as a whisper.

Why the Nose?

You might wonder, "Why the nose?"
Think of your nose as a tunnel right next to your mouth.

• When you whisper, you blow air through your nose. The nose picks up this "air turbulence" better than your throat does.
• Because the sensor is right there, it catches your voice before it even escapes into the room. It's like having a secret tunnel for your voice that no one else can hear.

The "Hand Cover" Trick

The paper also suggests a cool social hack. If you cover your mouth and nose with your hand while talking to the device:

1. Privacy: No one can lip-read you.
2. Signal: It acts like a universal sign to people around you: "I am talking to my computer, not ignoring you." It turns a weird gesture into a polite signal.

How Well Does It Work?

The researchers tested this in noisy places like trains and busy streets.

• Normal Microphones: Failed miserably in the noise.
• Apple AirPods (Voice Isolation): Great for normal talking, but they completely blocked out whispers (they thought the whisper was just noise).
• NasoVoce: It successfully heard the whispers and filtered out the train noise. It could understand you even when the world was chaotic.

The Bottom Line

NasoVoce is like giving your glasses a superpower. It allows you to have a continuous, private conversation with your AI assistant anywhere, anytime, without shouting, without wearing a mask, and without disturbing the people around you. It turns your glasses into a silent, always-on communication device that works even when the world is too loud to hear you.

Technical Summary

Here is a detailed technical summary of the paper "NasoVoce: A Nose-Mounted Low-Audibility Speech Interface for Always-Available Speech Interaction."

1. Problem Statement

The increasing integration of generative AI into daily life necessitates "always-available" voice interfaces. However, current voice interaction methods face four critical trade-offs that prevent seamless adoption:

• Wearability: Existing solutions (e.g., in-ear buds, face-mounted cameras for lip-reading, or intraoral sensors) are often uncomfortable for continuous use or socially intrusive.
• Silence/Privacy: Speaking aloud in public raises privacy concerns and social awkwardness. "Silent speech" (subvocalization) systems often lack vocabulary size or require complex sensors.
• Noise Robustness: Whispered speech, which offers a balance of privacy and vocabulary, is highly susceptible to environmental noise. Standard noise-canceling algorithms (e.g., in AirPods) often fail to distinguish whispers from background noise, treating them as interference.
• Vocabulary & Speed: Many silent speech interfaces are limited to small command sets or require slow, letter-by-letter input, making them unsuitable for natural, open-ended AI conversation.

The Gap: No existing system simultaneously satisfies wearability, silence, noise robustness, and full vocabulary support for continuous interaction.

2. Methodology: NasoVoce

The authors propose NasoVoce, a wearable interface mounted on the nasal bridge (specifically the nose pads of smart glasses).

A. Hardware Configuration

• Dual-Sensor Integration: The device integrates two MEMS sensors into the nose pad:
  1. MEMS Microphone: Captures air-conducted sound. It provides high-quality audio but is sensitive to environmental noise.
  2. MEMS Vibration Sensor (Syntiant V2S200D): Captures bone- and skin-conducted vibrations from the nasal bridge. It is highly robust to environmental noise but yields lower signal quality.
• Anatomical Advantage: The nasal bridge is uniquely positioned to capture:
  • Air-conducted signals: Due to proximity to the mouth.
  • Bone/Skin-conducted signals: Unlike throat microphones that rely on vocal cord vibrations (voiced signals), the nasal bridge captures aerodynamic turbulence and cavity resonance generated even during unvoiced whispering.
• Data Acquisition: Both sensors operate at 16kHz, 16-bit, and are time-synchronized via a TDM audio interface to eliminate latency mismatches. The vibration sensor is mapped to the left channel and the microphone to the right, creating a stereo stream compatible with standard audio interfaces.

B. Deep Learning Model: D-DCCRN

To fuse these complementary inputs, the authors developed D-DCCRN (Dual-DCCRN), an extension of the Deep Complex Convolution Recurrent Network (DCCRN).

• Architecture: The model processes both the real and imaginary components of the Short-Time Fourier Transform (STFT) for both the Microphone (Mic) and Vibration (Vib) signals simultaneously using complex-valued CNN layers and LSTM.
• Rationale: Whispered speech lacks harmonic structure and resembles noise. Standard magnitude-based networks struggle to separate it. D-DCCRN leverages phase information, which differs significantly between near-field whisper turbulence (captured by Vib) and far-field environmental noise.
• Training Strategy:
  • Dataset: 104 hours of paired data from 45 participants (clean speech + vibration) mixed with DEMAND noise dataset samples at various SNRs (-10dB to 10dB).
  • Loss Functions:
    1. Audio Enhancement Loss ($L_{ae}$): Mean Squared Error (MSE) and Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) to reconstruct clean speech.
    2. Knowledge Distillation Loss ($L_{kd}$): Uses OpenAI Whisper Large-v2 as a "teacher" model. The system minimizes the divergence between the ASR output of the enhanced noisy input and the clean input (using both hard token loss and soft probability distribution loss).

3. Key Contributions

1. Novel Sensor Placement: Demonstrated that the nasal bridge is an optimal location for capturing both air-conducted and bone/skin-conducted signals, specifically enabling the detection of whispered speech which is often missed by throat or skull-contact sensors.
2. Dual-Input Fusion Model: Introduced D-DCCRN, a complex-valued neural network that effectively fuses noisy microphone data with robust vibration data to reconstruct high-quality speech.
3. Comprehensive Evaluation: Validated the system across ASR accuracy (WER/CER), objective quality metrics (PESQ, STOI), and subjective user ratings (MUSHRA), proving its efficacy in both normal and whispered modes under noise.

4. Results

The system was evaluated against a baseline of Microphone-only (Mic), Vibration-only (Vib), and the Enhanced (D-DCCRN) output.

• ASR Accuracy (WER/CER):
  • Normal Speech: As noise increases, Mic accuracy drops sharply. Vib remains stable but has lower accuracy than Mic in quiet conditions. Enhanced output consistently outperforms Mic in noisy conditions (0dB and above) and matches or exceeds Vib in quiet conditions.
  • Whispered Speech: Mic fails to capture whispers in noise. Vib captures whispers but with lower accuracy due to weak vibration signals. Enhanced output significantly outperforms both, maintaining high recognition accuracy even in noisy environments.
• Objective Quality (PESQ/STOI):
  • The Enhanced signal consistently outperforms the raw Mic signal across all noise levels.
  • In extreme noise (10dB SNR), the Vib signal alone sometimes outperforms the Enhanced model, suggesting the microphone input may introduce artifacts when the SNR is critically low.
• Subjective Quality (MUSHRA):
  • Human raters preferred the Enhanced signal over Mic and Vib in low-to-moderate noise.
  • In very high noise (10dB), Vib was rated slightly higher than Enhanced, reinforcing the need for adaptive fusion.
• In-the-Wild Testing:
  • Tested in cafes, busy roads, walking, and trains.
  • Comparison with AirPods Pro 2: Apple's "Voice Isolation" feature successfully filtered noise for normal speech but completely suppressed whispered speech, treating it as background noise. NasoVoce successfully captured whispers in all environments.

5. Significance and Future Work

• Practicality: NasoVoce demonstrates the feasibility of a discreet, always-available AI interface that does not require speaking aloud, solving the privacy and social acceptability issues of current voice assistants.
• Whisper Recognition: It is one of the first systems to successfully recognize unrestricted vocabulary whispered speech in noisy environments, a capability critical for private AI interaction.
• Adaptive Fusion: The results suggest that a static fusion model is not optimal for all conditions. Future work proposes an SNR-adaptive gating mechanism that dynamically switches between Mic, Vib, or Fusion modes based on real-time noise levels to maximize performance.
• Form Factor: The design is specifically tailored for integration into smart glasses, leveraging the nose pad as a natural, unobtrusive mounting point.

In conclusion, NasoVoce bridges the gap between silent speech interfaces and robust, natural voice interaction, offering a viable path toward continuous, private, and context-aware AI companions.