MoXaRt: Audio-Visual Object-Guided Sound Interaction for XR

MoXaRt is a real-time Extended Reality system that leverages a cascaded audio-visual architecture to separate up to five concurrent sound sources, significantly improving speech intelligibility and reducing cognitive load in complex acoustic environments.

The Gist

Imagine you are at a bustling party. There are three different conversations happening at once, a band playing in the corner, and the clinking of glasses. Right now, your brain has to do the heavy lifting to figure out who is saying what, often leaving you feeling exhausted and unable to focus on the person you actually want to talk to. This is the "Cocktail Party Problem," and it gets even worse in Virtual Reality (XR) headsets, where you can't just turn your head to block out noise.

MoXaRt is a new system designed to solve this. Think of it as a personal audio DJ for your real life, but one that lives inside your glasses.

Here is how it works, broken down into simple concepts:

1. The "Super-Ears" and "Super-Eyes"

Most noise-canceling headphones just try to silence everything that isn't speech, or they try to silence everything that is speech. They are like a blunt hammer.

MoXaRt is different. It uses both your eyes and your ears.

• The Eyes: The system looks at the video feed from your glasses. It spots faces and instruments (like a guitar or piano). It essentially says, "Okay, I see a person talking over there, and I see a violinist over here."
• The Ears: It listens to the messy audio coming from your microphone.
• The Magic: It combines these two. Because it sees the violinist, it knows exactly which part of the messy audio belongs to the violin. It can then "pluck" that sound out of the mix, just like pulling a specific thread out of a tangled ball of yarn.

2. The "Audio Mixer" in Your Hand

Once MoXaRt has separated the sounds, it gives you a virtual mixing board.

• The Concert: Imagine you are at a live show. You love the cello, but the piano is too loud. With MoXaRt, you can point at the cellist and say (or use a slider), "Turn the cello up!" and "Turn the piano down." You get to be the sound engineer for your own experience.
• The Busy Cafe: You are trying to talk to a friend, but a loud band is playing nearby. You can look at your friend, and the system will boost their voice while fading the band into the background. It's like having a spotlight on your conversation partner while the rest of the room fades into a soft blur.

3. How Fast Is It?

You might wonder, "Does this take too long to process?"
Currently, there is a tiny delay—about 2 seconds. Think of it like a live TV broadcast with a slight lag. The system looks at a 1-minute chunk of video and audio, figures out who is making what sound, and then plays the "remixed" version back to you. While 2 seconds isn't instant, it's fast enough to make a huge difference in understanding speech and enjoying music.

4. Why Does This Matter?

The researchers tested this with real people in noisy situations (like crowded meetings and concerts).

• The Result: People understood what others were saying 36% better when using MoXaRt compared to not using it.
• The Feeling: People felt much less mentally tired (cognitive load) because they didn't have to strain their brains to filter out the noise.

The Big Picture

Before MoXaRt, if you wanted to hear one person in a noisy room, you had to shout, move closer, or wear headphones that blocked out everything.

MoXaRt changes the rules. It turns the chaotic, tangled mess of the real world into a clean, organized playlist that you can control. It doesn't just silence the world; it lets you curate it, giving you the power to focus on what matters and ignore what doesn't.

In short: MoXaRt is like giving you a remote control for reality, allowing you to turn up the volume on the things you love and turn down the things that annoy you, all in real-time.

Technical Summary

Here is a detailed technical summary of the paper "MoXaRt: Audio-Visual Object-Guided Sound Interaction for XR."

1. Problem Statement

Extended Reality (XR) users often face "acoustic overwhelm" in complex environments where multiple sound sources (speech, music, noise) are entangled. Unlike visual elements, which can be digitally manipulated, ambient audio remains a mixed, unfiltered stream. This leads to:

• Reduced Scene Awareness: Important sounds (e.g., a specific conversation or solo) are drowned out.
• High Cognitive Load: Users struggle to filter noise, impairing social engagement and comprehension.
• Technical Limitations: Existing solutions rely on bulky hardware (large microphone arrays) or computationally expensive offline models. Furthermore, current audio-visual separation models are typically passive, offline, or limited to binary filtering (target vs. noise), lacking the ability to dynamically remix multiple concurrent sources in real-time on wearable devices.

2. Methodology: MoXaRt System

MoXaRt is a real-time XR system that enables fine-grained, interactive control over a user's soundscape by separating sound sources from a single audio channel using audio-visual cues.

A. System Architecture

The system employs a cascaded, coarse-to-fine architecture operating in parallel:

1. Coarse Sound Separation (Audio-Only):
   • A dilated convolutional network (based on AudioScopeV2) performs an initial separation of the raw mono audio into three general categories: Speech, Music, and Noise.
   • This stage is purely audio-based to minimize computational overhead and allow for pre-training on large-scale datasets (e.g., YFCC100M) using self-supervised techniques.
2. Visual Anchor Detection:
   • Face Detection: Uses a YOLOv8-based model to identify speakers and their spatial coordinates.
   • Instrument Detection: Uses a DeepLabv3+ model (MobileNetV2 backbone) to detect specific musical instruments (Vocal, Piano, Violin, Cello, Bass, Guitar, Flute).
3. Refinement Networks (Visual-Guided):
   • Speech Refinement: Visual anchors (faces) guide a refinement network to isolate individual speakers from the coarse speech track.
   • Music Refinement: Detected instruments dynamically activate a Teacher-Student distillation pipeline. A high-fidelity "Teacher" ensemble of Band Split Roformer (BSR) models generates pseudo-targets for specific instruments. A lightweight "Student" model is trained to mimic the teacher in real-time, outputting isolated instrument stems.

B. Implementation & Pipeline

• Hardware: Meta Quest 3 headset for capture; external Rode Wireless GO II microphone for mono audio input; NVIDIA RTX 5080 GPU on a PC for inference.
• Data Transmission: Uses WebRTC for bidirectional streaming (video/audio to PC, processed audio back to headset).
• Processing Pipeline: Operates in a 3-stage loop (Capture → Inference → Playback) with a 1-minute rolling buffer for context and a 1-second chunk size for processing.
• Latency: The system achieves a consistent end-to-end latency of ~2 seconds.
• User Interface: Visualizes detected sources as spatial markers. Users can adjust volume sliders for individual sources via controllers or hand gestures to remix the audio in real-time.

3. Key Contributions

1. MoXaRt System: The first XR system to integrate real-time, audio-visual sound separation as a core primitive for direct, interactive control over authentic sounds.
2. Cascaded Audio-Visual Transformer Model: A novel architecture that leverages visual anchors (faces/instruments) to guide refinement networks, enabling the separation of up to 5 concurrent sources (e.g., 2 voices + 3 instruments) from a single audio channel.
3. New Dataset: A curated dataset of 30 one-minute recordings featuring complex mixtures of up to 5 speakers and 3 instruments, designed specifically for evaluating interactive sound separation.
4. Teacher-Student Distillation: A method to transfer high-fidelity separation capabilities from heavy, offline ensemble models to lightweight, real-time student models suitable for XR.

4. Results

Technical Evaluation (Objective)

Evaluated on the new dataset against state-of-the-art baselines (Sound of Pixels, DAVIS, AudioScopeV2, AV-Mossformer2):

• Speech Intelligibility: MoXaRt (Offline) achieved the lowest Word Error Rate (WER) of 0.3824, outperforming AV-Mossformer2 (0.3956).
• Real-Time Performance: The distilled Real-Time model achieved a WER of 0.4990, significantly outperforming its architectural basis, AudioScopeV2 (0.5263), while maintaining comparable perceptual quality (DNSMOS scores).
• Separation Capability: Successfully separated up to 4 distinct human voices or a mix of 5 total sources in real-time.

User Study (Subjective & Behavioral)

A within-subjects study with 22 participants across 6 scenarios (e.g., noisy dining, multi-speaker conferences, orchestra evaluation).

• Listening Comprehension: Participants using MoXaRt showed a 36.2% increase in listening comprehension scores compared to the baseline ($p < 0.01$). The improvement was most pronounced in difficult, noisy conditions.
• Cognitive Load & Clarity: Users reported significantly higher clarity and lower cognitive load ($p < 0.001$).
• Subjective Ratings: Overall experience ratings were significantly higher for MoXaRt (Mean 9.11 vs. 3.73 for baseline).

5. Significance and Future Implications

• Paradigm Shift: Moves XR audio from passive rendering to active, object-centric manipulation, allowing users to act as their own "audio engineers."
• Accessibility: Offers a powerful solution for the "cocktail party problem," aiding individuals with hearing difficulties in noisy social environments.
• AI Integration: The separated, speaker-tagged audio streams serve as clean inputs for downstream AI applications, such as real-time multi-person translation and queryable Retrieval-Augmented Generation (RAG) for meeting summaries.
• Limitations & Future Work: Current limitations include reliance on line-of-sight (performance degrades if sources are occluded) and the need for a tethered PC for inference. Future work aims to optimize for standalone headsets, implement audio-only fallbacks, and explore gaze-based or neural interfaces for more natural interaction.

In conclusion, MoXaRt demonstrates that real-time, visually-guided source separation is feasible and highly effective, paving the way for more perceptive and socially adept XR experiences.