SUBARU: A Practical Approach to Power Saving in Hearables Using SUB-Nyquist Audio Resolution Upsampling

The paper proposes SUBARU, a power-efficient framework for hearables that intentionally employs sub-Nyquist sampling and low bit-resolution ADCs to achieve a 3.31x reduction in power consumption while maintaining high-quality multimodal speech enhancement through a novel wideband reconstruction methodology.

The Gist

Imagine you are wearing a pair of high-tech earbuds (called "hearables") in a very noisy place, like a busy bus or a crowded classroom. You want to hear a friend's voice clearly, but the background noise is drowning them out.

Usually, to make your voice clear, these earbuds need to work hard. They record sound at a very high speed and with extreme detail (like a 4K camera recording video). This requires a lot of battery power, which kills your earbuds' battery life quickly.

Enter SUBARU.

Think of SUBARU as a clever "split-personality" system that solves the battery problem without sacrificing sound quality. Here is how it works, using some simple analogies:

1. The "Low-Res Sketch" Strategy (The Earbuds)

Normally, your earbuds try to record a perfect, high-definition "photograph" of the sound. This is heavy and drains the battery.

SUBARU changes the game. Instead of taking a high-definition photo, the earbuds take a quick, low-resolution sketch.

• The Analogy: Imagine you are trying to send a picture of a cat to a friend. Instead of sending a massive 100MB high-res photo that takes forever to upload and drains your phone, you quickly draw a simple 5MB sketch of the cat.
• The Result: The earbuds record the sound at a much slower speed and with less detail. This is like turning down the "quality" knob on a camera. Because they are doing less work, they use 3.3 times less battery power. Your earbuds can now last much longer.

2. The "Magic Restorer" (The Phone)

Here is the catch: A low-res sketch of a cat doesn't look very good on its own. It's blurry and missing details.

This is where your smartphone comes in. The earbuds send that tiny, blurry sketch to your phone via Bluetooth. Your phone is much more powerful than the earbuds (it has a bigger battery and a stronger brain).

• The Analogy: Your phone acts like a super-powered AI artist. It receives your simple sketch and instantly fills in all the missing details, colors, and textures to recreate a perfect, high-definition photo of the cat.
• The Tech: The paper calls this "Sub-Nyquist Audio Resolution Upsampling." In plain English, it's a smart algorithm that guesses the missing high-pitched sounds (like the "s" in "snake" or the crispness of a cymbal) that were lost when the earbuds recorded the "sketch."

3. The "Bone Conduction" Secret Sauce

To make this even better, SUBARU uses two types of microphones:

1. Air Microphone: Listens to the air (like normal).
2. Bone Microphone: Listens to the vibrations in your jawbone when you speak.

• The Analogy: Imagine you are in a storm. The air microphone hears the wind howling (noise). But the bone microphone feels the vibrations of your voice through your jaw, which the wind can't shake.
• The Result: The system combines the "sketch" from the air mic with the "vibration sketch" from the bone mic. This helps the phone's "AI artist" separate your friend's voice from the background noise much better than before.

4. Why It's a Game Changer

• Battery Life: Because the earbuds are just doing a quick "sketch" instead of a "4K video," they save a massive amount of energy. The paper says this could triple the battery life of your hearables.
• Real-Time Chat: The phone is so fast at "restoring" the sketch that it happens in less than a blink of an eye (about 71 milliseconds). This means you can talk to someone in real-time without that annoying "robot delay" you sometimes get on video calls.
• Quality: Even though the earbuds recorded a "sketch," the final result on your phone sounds just as clear as if the earbuds had recorded a "4K video" the whole time.

Summary

SUBARU is like a smart team effort:

• The Earbuds are the lightweight scouts. They gather information quickly and efficiently to save energy, sending back a rough draft.
• The Phone is the heavy lifter. It takes that rough draft and polishes it into a masterpiece, removing noise and filling in the missing details.

This allows you to wear your hearables all day long with a full battery, while still hearing crystal-clear audio even in the noisiest environments.

Technical Summary

Here is a detailed technical summary of the paper "SUBARU: A Practical Approach to Power Saving in Hearables Using Sub-Nyquist Audio Resolution Upsampling."

1. Problem Statement

Hearables (wearable computers like earbuds) typically rely on Air Conduction Microphones (ACMs) and Bone Conduction Microphones (BCMs) for multimodal speech enhancement (SE) in noisy environments. However, existing solutions face critical limitations regarding power efficiency and resource constraints:

• High Power Consumption: Current systems sample audio at Nyquist rates (>16 kHz) and high bit resolutions (12–24 bits) using Analog-to-Digital Converters (ADCs). This consumes significant battery power, limiting the lifespan of hearables.
• Inefficient Reconstruction: Existing Sub-Nyquist approaches often lack robust wideband reconstruction methodologies. They struggle to recover high-frequency components and clean phases from low-resolution, narrowband signals, especially in noisy conditions.
• Model Complexity: State-of-the-art (SOTA) models often use Generative Adversarial Networks (GANs) which, while high-quality, have massive parameter counts (hundreds of millions) and high inference latency, making them unsuitable for real-time streaming on mobile platforms connected to hearables.
• Lack of Joint Optimization: No existing framework jointly optimizes low-power sampling (sub-Nyquist frequency and low bit depth) with multimodal SE and Bandwidth Extension (BWE).

2. Methodology: SUBARU

The authors propose SUBARU (Sub-Nyquist Audio Resolution Upsampling), a split-architecture framework designed to shift the computational burden from the hearable device to the connected mobile platform (e.g., smartphone).

A. System Architecture

• Hearable Side (Low Power): ACMs and BCMs sample audio at sub-Nyquist frequencies (e.g., 4 kHz) and low bit resolutions (e.g., 8-bit). This drastically reduces ADC power consumption.
• Mobile Side (High Compute): The low-resolution, noisy signals are transmitted via Bluetooth to a mobile device. SUBARU runs on the mobile device to perform joint Bandwidth Extension (BWE) and Multimodal Speech Enhancement (SE), reconstructing high-fidelity, wideband audio (e.g., 16–24 kHz).

B. Model Architecture (U-Net based Hybrid)

SUBARU utilizes a hybrid architecture combining waveform-based and spectrum-based methods within a U-Net framework, consisting of four key modules:

1. Spectral Enhancement Network (SEN): Takes the spectrogram of the noisy, low-resolution ACM signal. It uses a 2D U-Net with a Mamba block (a state-space model) as the bottleneck to capture global correlations efficiently, replacing heavy Transformer blocks to reduce parameters.
2. Upsampling Network: Inspired by HiFi-GAN, this module converts the enhanced spectrogram back into a raw waveform using transposed convolutions (256x upsampling ratio).
3. Time Enhancement Network: A 1D U-Net that fuses features from the ACM (acoustic) and BCM (vibration/accelerometer) modalities. It refines the waveform in the time domain, leveraging the less noisy nature of BCM signals.
4. Amplitude-Phase Enhancement Network: Operates in the frequency domain to reconstruct clean phases and amplitudes. It addresses phase wrapping issues using specific loss functions to ensure high perceptual quality in noisy conditions.

C. Training and Loss Functions

SUBARU employs a multi-domain training strategy (time, phase, and frequency) using four specific loss functions:

• Multi-scale Loss: Minimizes error between downsampled versions of the enhanced and ground-truth waveforms.
• Multi-period Loss: Enforces periodicity consistency in the waveform.
• Phase-Spectrum Loss: Uses instantaneous phase and group delay anti-wrapping losses to reconstruct clean phases from noisy inputs.
• Multi-resolution STFT Loss: Ensures accurate frequency content reconstruction across different resolutions.

3. Key Contributions

• Sub-Nyquist Sampling Strategy: Demonstrates that reducing ADC sampling rates to 4 kHz and bit depth to 8 bits on hearables can save significant power without permanent audio quality loss, provided a robust reconstruction algorithm exists.
• Efficient Hybrid Architecture: Introduces a U-Net-based model that integrates Mamba for efficient sequence modeling, achieving GAN-like perceptual quality with significantly fewer parameters (3.61M vs. hundreds of millions in GANs).
• Multimodal Integration: Successfully fuses ACM and BCM data (vibration and accelerometer) for noise suppression and bandwidth extension, a capability missing in many SOTA single-modal or narrowband models.
• Real-Time Streaming Capability: Achieves ultra-low inference latency (1.74 ms on GPU, 71 ms on mobile), enabling frame-by-frame streaming that meets ITU standards for voice communication (<150 ms latency).

4. Experimental Results

The authors evaluated SUBARU against 10 baselines (including GANs like HiFi++, AERO, SEANet, and U-Nets like TFiLM) on speech (VCTK) and music (MagnaTagATune) datasets.

• Power Savings: By switching from {24 kHz, 12-bit} to {4 kHz, 8-bit} sampling on the hearable, SUBARU achieves a 3.31x reduction in power consumption, theoretically extending hearable battery life by the same factor.
• Audio Quality:
  • SUBARU outperforms SOTA U-Nets and matches or exceeds GANs in metrics like LSD (Log-Spectral Distance), PESQ, and STOI.
  • It achieves a PESQ of 3.00 and STOI of 0.90 on noisy speech, comparable to high-complexity GANs.
• Efficiency & Latency:
  • Inference Time: 1.74 ms on a desktop GPU; 71 ms on a Google Pixel7 mobile phone.
  • Comparison: SUBARU is 3.61x faster than HiFi++ and 20.68x faster than AERO on desktops.
  • Streaming: The 71 ms inference time on mobile is well within the 150 ms threshold for acceptable voice call latency, enabling real-time streaming.
• Robustness: Evaluated in diverse live noise scenarios (bus rides, classrooms, cars), SUBARU significantly reduced Character Error Rate (CER) compared to raw signals, demonstrating effectiveness in real-world environments.

5. Significance

SUBARU represents a paradigm shift in hearable design by decoupling the sampling constraints from the processing constraints.

• Practicality: It solves the "power vs. quality" trade-off, allowing hearables to operate for much longer on a single charge while maintaining high-fidelity audio through mobile-side processing.
• Scalability: The model's small footprint (13.77 MB) and low latency make it deployable on current consumer smartphones, bridging the gap between theoretical low-power research and commercial viability.
• Future Impact: This approach paves the way for next-generation hearables that can support complex multimodal AI applications (health monitoring, AR) without being bottlenecked by battery life, as the heavy lifting is offloaded to the user's phone.