Fine-grained Soundscape Control for Augmented Hearing

This paper introduces Aurchestra, a novel system for resource-constrained hearables that enables real-time, fine-grained control over up to five overlapping sound sources by combining a dynamic interface with an optimized on-device multi-output extraction network, effectively transforming the acoustic environment into a programmable mix.

The Gist

Imagine your ears are like a radio that can only do two things: either play everything at full volume, or mute everything except one specific station. That's how current hearing aids and noise-canceling headphones work. If you're in a busy park, you can either hear the birds or the traffic, but you can't hear the birds loudly while turning the traffic down to a whisper.

Enter Aurchestra (a clever mix of "Audio" and "Orchestra"). This new system turns your hearing into a live mixing console, letting you conduct the symphony of your daily life.

Here is how it works, broken down into simple concepts:

1. The Problem: The "All-or-Nothing" Switch

Think of your current hearing experience like a light switch. You can flip it ON (hear everything, including annoying construction noise) or OFF (mute everything, including your friend talking to you).

• The Reality: The real world isn't a light switch; it's a jungle of sounds. You might want to hear a baby crying (safety), a dog barking (curiosity), and a car honking (danger), while simultaneously turning down the hum of a vacuum cleaner.

2. The Solution: The "Audio DJ"

Aurchestra acts like a personal DJ for your brain. Instead of just one volume knob for the whole world, it gives you a separate volume slider for every type of sound happening around you.

• The Analogy: Imagine you are at a party. Usually, you hear the music, the clinking glasses, and the chatter all mashed together. Aurchestra lets you say, "I want the music loud, the clinking glasses quiet, and the chatter just right." It separates the sounds into individual "tracks" so you can mix them yourself.

3. How It Works (The Magic Behind the Scenes)

Step A: The "Eagle-Eyed" Detective (Sound Detection)
Before you can mix the music, you need to know what instruments are playing. Aurchestra uses a smart AI on your phone to listen to the world and say, "Hey, I hear a siren, a dog, and someone talking right now."

• The Innovation: Old systems would show you a long, boring list of 200 possible sounds (like a phone book). Aurchestra is smart enough to only show you the active sounds. It's like a menu that only lists the dishes currently being cooked in the kitchen, rather than the entire menu of the restaurant.

Step B: The "Magic Filter" (The Neural Network)
Once you decide what you want to hear (e.g., "Keep the speech, mute the traffic"), the system runs a tiny, super-fast computer program right on your earbuds.

• The Challenge: Doing this math is usually heavy, like trying to run a video game on a calculator. The researchers built a "lightweight" version of this AI that fits on tiny, low-power chips in hearing aids without draining the battery.
• The Speed: It works so fast (in less than 10 milliseconds) that you don't even notice a delay. It feels instant, like magic.

Step C: The "Mixing Board" (Your Control)
You tap the sounds you want on your phone, and you get individual volume sliders.

• Scenario: You are walking down a busy street.
  • Traffic: Turn the volume down to 20% (so you don't get distracted).
  • Speech: Turn the volume up to 100% (so you can hear your friend).
  • Birds: Keep them at 50% (enjoy the nature).
  • Result: You walk through the city hearing a perfectly balanced, personalized soundscape.

4. Why This Matters

• Safety: You can hear a car horn or a siren clearly even if you are wearing noise-canceling headphones.
• Focus: You can tune out the annoying hum of an air conditioner to focus on a conversation.
• Accessibility: For people with hearing loss, it's not just about making things louder; it's about making the right things clearer while hiding the noise that hurts their ears.

The Bottom Line

Aurchestra changes hearing from a passive experience (just listening to whatever happens) into an active creative process. It treats the world not as a chaotic noise storm, but as a programmable studio where you are the conductor, deciding exactly which instruments get the spotlight and which ones take a backseat.

In short: It turns your ears from a simple microphone into a smart, customizable sound mixer for your entire life.

Technical Summary

Here is a detailed technical summary of the paper "Fine-grained Soundscape Control for Augmented Hearing" (Aurchestra).

1. Problem Statement

Current hearable devices (e.g., smart earbuds, hearing aids) offer only "blunt" sound controls: either global noise cancellation (suppressing everything) or focusing on a single target sound (e.g., speech). Real-world acoustic environments are complex "orchestras" containing simultaneous sources (speech, traffic, birds, alarms, etc.). Users often need to independently adjust the volume of specific sound classes (e.g., amplify a siren while suppressing traffic noise) without losing other ambient sounds. Existing systems lack the ability to:

• Extract multiple target sound classes simultaneously in real-time.
• Provide independent, per-class volume control (mixing).
• Adapt dynamically to the changing acoustic environment without forcing users to navigate static, overwhelming lists of categories.
• Operate within the strict latency (<20ms) and power constraints of resource-constrained hearable hardware.

2. Methodology

The authors propose Aurchestra, a system comprising three core components: a real-time multi-output extraction network, hardware-specific model optimizations, and a dynamic user interface.

A. Real-Time Multi-Output Sound Extraction

• Architecture: The system uses a time-frequency (TF) domain neural network. It employs a dual-path design with a spectral stage (modeling frequency sequences) and a temporal stage (modeling time sequences).
• Conditioning: The network is conditioned on a multi-hot encoding of user-selected sound classes using Feature-wise Linear Modulation (FiLM) layers. This allows the model to dynamically route specific classes to specific output streams.
• Output Strategy: Instead of generating $K$ output streams for all $K$ trained classes (which is computationally wasteful), the system generates a fixed, small number of output streams ($O$, e.g., 5). The network learns to map the selected target classes to these $O$ streams based on their alphabetical order in the multi-hot vector. This avoids the need for Permutation Invariant Training (PIT) and stitching algorithms.
• Latency Optimization: To achieve low latency, the system uses a dual-window STFT approach with a 6ms chunk size, 4ms lookahead, and 6ms lookback. By zero-padding the lookback window and recomputing the synthesis window, the algorithmic latency is reduced to 10ms (vs. standard 16ms+).

B. Hardware-Specific Model Optimizations

The system is optimized for three distinct hardware platforms to ensure real-time performance:

1. Orange Pi 5B (High-End): Uses a standard architecture with bidirectional LSTMs and 3x3 causal convolutions.
2. Raspberry Pi 4B (Mid-Range): Compresses the frequency domain using strided convolutions and reduces latent dimensions to fit lower compute power.
3. NeuralAids (GreenWaves GAP9 AI Accelerator): Replaces sequential LSTMs with highly parallelizable MLP-Mixer blocks to leverage the chip's 10-core parallel processing capabilities. It utilizes 8/16-bit fixed-point operations and removes layer normalization for efficiency.

C. Dynamic Interface & Sound Event Detection (SED)

• Problem: Static lists of 20+ categories are cognitively burdensome.
• Solution: A companion smartphone app runs a lightweight Sound Event Detection (SED) model that periodically scans the environment.
• Model: Based on the Audio Spectrogram Transformer (AST). The authors fine-tuned AST on heavily overlapped mixtures (1–5 simultaneous sources) to improve robustness where standard models fail.
• Interaction: The interface only displays active sound classes detected in the current environment. Users can tap these to adjust volume sliders independently.
• Latency Mitigation: To hide the 3–5 second inference time required for accurate SED, the system uses a staggered buffering strategy, processing the previous audio chunk while recording the current one, making the interface feel responsive.

3. Key Contributions

1. First Fine-Grained Soundscape Control: Aurchestra is the first system to enable real-time, independent volume mixing of multiple sound classes on resource-constrained hearables.
2. Efficient Multi-Output Architecture: Introduced a dynamic mapping strategy that limits output streams to a small number (e.g., 5) while supporting any subset of target classes, outperforming fixed-output strategies in both efficiency and accuracy.
3. Hardware-Aware Design: Demonstrated real-time inference (processing 6ms chunks) on diverse platforms (Orange Pi, Raspberry Pi, and GAP9 AI accelerator) with power consumption as low as 56 mW.
4. Robust Overlapping Detection: Achieved 93.2% accuracy in detecting 5 simultaneous overlapping sound sources using fine-tuned AST, significantly outperforming baselines (YAMNet at 63.8%, standard AST at 81.5%).

4. Results

• Sound Extraction Performance:
  • SNR Improvement: Achieved 11.99 dB Signal-to-Noise Ratio improvement (SNRi) on the Orange Pi platform, outperforming the prior state-of-the-art (Waveformer/Semantic Hearing) which achieved 7.29 dB, while using less than half the parameters (0.5M vs 1.2M).
  • Multi-Target Stability: The system maintains stable performance when extracting up to 5 simultaneous streams. Performance degrades significantly only when attempting to extract 20 streams simultaneously.
• Hardware Performance:
  • Inference Time: All models process 6ms audio chunks in real-time with mean inference times of 5.22 ms (Orange Pi), 4.47 ms (Raspberry Pi), and 5.23 ms (GAP9).
  • Power: The NeuralAids model consumes 56 mW at 330 MHz, proving feasibility for battery-powered hearables.
• User Studies:
  • Listening Quality: In blind listening tests ($N=17$), Aurchestra improved background noise suppression by +1.54 points and overall listening experience by +0.95 points (on a 5-point scale) compared to a no-AI baseline, with no noticeable distortion.
  • Interface Usability: The dynamic interface reduced the time required for users to select target sounds by 67.9% compared to a static list interface.
  • Per-Class Clarity: Impulsive sounds (alarms, door knocks) were isolated with high clarity (>4.2/5), while complex sounds (bird chirps, typing) scored lower but remained acceptable (>3.5/5).

5. Significance

Aurchestra fundamentally shifts the paradigm of augmented hearing from binary filtering (on/off) to programmable audio. By treating the real world as a mixable studio, it empowers users to sculpt their auditory environment, enhancing safety (amplifying alarms/traffic), accessibility (focusing on speech), and comfort (suppressing distractions). The work demonstrates that complex, multi-class audio separation and mixing are feasible on low-power, edge devices, paving the way for a new generation of intelligent, context-aware hearables that adapt dynamically to user needs and environmental contexts.