💻 computer science

Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification

The paper proposes R $^{2}$ ScP, a novel framework that addresses performance degradation in incomplete Audio-Visual Question Answering by shifting from generative imputation to a retrieval-based recovery approach enhanced by semantic-consistent purification and two-stage training to effectively restore missing modality knowledge.

Original authors: Jiayu Zhang, Shuo Ye, Qilang Ye, Zihan Song, Jiajian Huang, Zitong Yu

Published 2026-04-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiayu Zhang, Shuo Ye, Qilang Ye, Zihan Song, Jiajian Huang, Zitong Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery, but you've lost half your clues. Maybe the audio recording of a crime scene is corrupted, or the video footage is blurry. This is the daily struggle for computers trying to answer questions about videos and sounds (a task called Audio-Visual Question Answering).

Most current computer programs try to fix this by guessing. If they can't hear the sound, they try to "hallucinate" or invent what the sound might have been based on what they see. It's like a detective trying to guess the sound of a specific violin solo just by looking at a picture of a concert hall. They might guess "music," but they'll likely get the specific notes or the unique tone of that violin wrong. This leads to confusion and wrong answers.

The Paper's Big Idea: "Retrieving to Recover" (R2ScP)

Instead of guessing, the authors propose a smarter approach: Go to the Library.

Think of their new system, R2ScP, as a detective who, when a clue is missing, doesn't try to invent it. Instead, they run to a massive, organized library of real-world examples (a database) to find a real recording that matches the current scene.

Here is how it works, broken down into simple steps:

1. The Library Search (Cross-Modal Retrieval)

Imagine you are watching a video of a cat playing a piano, but the audio is missing.

Old Way: The computer tries to imagine a generic "piano sound."
R2ScP Way: The computer looks at the video of the cat, goes to its "Library of Real Sounds," and searches for: "What does a cat playing a piano actually sound like?" It pulls up a real audio clip from the library that matches the visual scene.

2. The Quality Control (Context-Aware Purification)

Here is the catch: The library is huge, and sometimes it pulls up the wrong thing. Maybe it finds a recording of a dog barking because the library thinks "cat" and "dog" are similar animals. Or maybe it finds a piano sound, but it's from a jazz song, while your video is a classical piece.

This is where the Purification step comes in. Think of this as a strict editor or a bouncer.

The system takes the audio it found in the library.
It checks it against the video and the question you asked.
The Editor says: "This part sounds like a dog? Cut it! This part is jazz, but the video is classical? Cut it! But this specific note matches the cat's paw movement? Keep it!"
It surgically removes the "noise" (the wrong stuff) and keeps only the "signal" (the perfect, real-world sound).

3. The Team of Experts (Mixture of Experts)

Finally, the system uses a team of specialists to solve the puzzle.

There is a Visual Expert (who looks at the video).
There is a Text Expert (who reads your question).
There is a Recovered Audio Expert (who uses the cleaned-up sound from the library).

These experts vote on the answer. Because the audio expert is now using real data that was carefully checked, the team is much more confident and accurate than if they were just guessing.

Why is this better?

No Hallucinations: The computer isn't making things up; it's finding real facts.
Specific Details: It doesn't just know "music"; it knows the specific timbre of that specific instrument in that specific video.
Robustness: Even if the data is very broken (like 70% missing), this system keeps working because it can pull in fresh, real examples from its library to fill the gaps.

In a nutshell:
Instead of a computer trying to imagine what a missing sound looks like (which often leads to mistakes), this new method looks it up in a database of real life and then cleans it up to make sure it fits perfectly. It's the difference between a detective guessing a suspect's voice and a detective playing back a real recording of that voice found in the archives.

1. Problem Statement

Audio-Visual Question Answering (AVQA) requires models to reason across visual, audio, and textual modalities to answer questions about dynamic scenes. While recent methods have advanced significantly, they suffer from a critical limitation: they assume modality completeness.

In real-world scenarios (e.g., sensor occlusion, device malfunction, or transmission failure), critical modalities (like audio in a concert video) are often missing. Existing solutions primarily rely on generative imputation (using GANs, diffusion models, or autoencoders to synthesize missing features). However, the authors argue that generative methods have intrinsic flaws:

Semantic Hallucination: They tend to generate generic, "common knowledge" features (e.g., a generic "music" sound) rather than specific, instance-level details (e.g., the specific timbre of a violin).
Noise Accumulation: The synthesized features often introduce semantic noise that confuses the reasoning process, leading to degraded performance in complex, unconstrained scenes.

2. Methodology: R2ScP Framework

The authors propose R2ScP (Retrieving to Recover via Semantic-consistent Purification), a paradigm shift from generating missing data to retrieving and purifying real-world semantic knowledge. The framework consists of four core components:

A. Cross-Modal Retrieval (CMR)

Instead of synthesizing features, R2ScP constructs an external memory bank $B$ using a unified semantic space (e.g., ImageBind).

Mechanism: When a modality is missing (e.g., audio), the available modality (e.g., visual) serves as a query ( $Q_{avl}$ ).
Process: The system calculates cosine similarity between the query and keys in the memory bank to retrieve the top- $n$ candidate feature segments ( $R$ ) that are semantically coherent with the context.

B. Context-aware Adaptive Purification (CAP)

Raw retrieval inevitably introduces irrelevant information (e.g., retrieving "applause" when the visual cue is a solo violin). To address this, CAP acts as a dynamic semantic filter with three phases:

Consistency-based Noise Profiling: It computes a "dissonance score" between retrieved tokens and the global context of the available modality. Tokens with high dissonance (inconsistent with the scene) are flagged as noise ( $\Omega_{noise}$ ).
Text-Guided Semantic Acquisition: Using the textual question as a high-level instruction, the model employs cross-attention and self-attention to identify the most salient, question-relevant features ( $\Omega_{salient}$ ) from the common knowledge pool.
Selective Feature Injection: The system performs a "surgical" replacement. It overwrites the identified noisy tokens with the high-quality, question-relevant semantic cues while preserving the consistent parts of the retrieved data. This ensures high-fidelity reconstruction.

C. Two-Stage Expert Training

To model the reliability of different information sources (original vs. recovered), R2ScP employs a Mixture of Experts (MoE) strategy with a decoupled curriculum:

Stage I (Expert Pre-training): Independent experts (Visual, Audio, Text) are trained on complete data to learn discriminative, modality-specific representations without relying on cross-modal shortcuts.
Stage II (Expert Mixing): The experts are frozen, and a Gating Network (Router) is trained to dynamically assign weights to the experts based on the input context. This allows the model to adaptively rely on the retrieved (purified) modality when the original is missing.

D. Optimization

The framework uses a compound loss function including standard cross-entropy and a Semantic Ranking Loss ( $L_{rank}$ ). This loss enforces a structured semantic manifold where:

Ground Truth > Retrieved Positive Samples > Retrieved Negative Samples.
This ensures the purified features are semantically superior to random noise but distinct from the ground truth, aiding the QA reasoning process.

3. Key Contributions

Paradigm Shift: Moves AVQA missing modality handling from generative imputation to retrieval-based recovery, effectively preserving modality-specific details that generative models often hallucinate.
Context-aware Adaptive Purification (CAP): A novel mechanism that dynamically filters semantic noise from retrieved data by enforcing consistency with available modalities and the user's question, ensuring high-fidelity feature reconstruction.
Robust Performance: Demonstrates that retrieving real-world knowledge is more effective than synthesizing pseudo-features for complex reasoning tasks.

4. Experimental Results

The authors evaluated R2ScP on Music-AVQA and AVQA datasets under various missing modality settings.

State-of-the-Art Performance: R2ScP achieved new SOTA average accuracies of 71.54% on Music-AVQA and 76.35% on AVQA, outperforming competitors like IMOL, SimMLM, and Missing-AVQA.
Robustness to Data Loss: In experiments with increasing missing rates (30% to 70%), R2ScP maintained superior stability. While generative baselines showed steep performance degradation as context became scarce, R2ScP's reliance on external knowledge allowed it to bridge the information gap effectively.
Ablation Studies:
- Removing the CAP module caused significant performance drops, confirming its necessity for noise filtering.
- Removing the Two-Stage Training (specifically the expert mixing stage) resulted in a massive drop (from 70.72% to 64.21%), highlighting the importance of the gating mechanism in handling incomplete inputs.
Visualization: t-SNE analysis showed that R2ScP produces tighter clustering and clearer separation between answer categories compared to baselines, indicating better semantic alignment.

5. Significance and Future Work

Significance:
This work challenges the dominance of generative models in incomplete multimodal learning. It proves that for tasks requiring fine-grained reasoning (like identifying specific instrument timbres), retrieving real-world evidence is superior to hallucinating synthetic features. The proposed CAP mechanism offers a robust solution for real-world deployment where sensor failures are common.

Limitations & Future Directions:

Inference vs. Training: The current model handles missing modalities primarily during inference. Future work aims to extend robustness to the training phase where data may be incomplete.
Retrieval Quality: Performance is sensitive to the quality and domain consistency of the retrieval corpus. Future research will explore larger-scale, domain-adaptive retrieval databases to further enhance generalization.