🤖 AI

Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought

The paper proposes "Separate First, Fuse Later" (SFFL), a novel audio-visual reasoning framework that mitigates cross-modal interference and hallucinations in Audio-Visual Large Language Models by enforcing modality-specific chain-of-thought reasoning followed by evidence fusion, resulting in significant accuracy and robustness improvements on AVQA benchmarks.

Original authors: Xuanchen Li, Yuheng Lu, Chenrui Cui, Tianrui Wang, Zikang Huang, Yu Jiang, Long Zhou, Longbiao Wang, Jianwu Dang

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xuanchen Li, Yuheng Lu, Chenrui Cui, Tianrui Wang, Zikang Huang, Yu Jiang, Long Zhou, Longbiao Wang, Jianwu Dang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a mystery by watching a video and listening to the audio at the same time. Usually, your brain does this effortlessly. But for Artificial Intelligence (AI), this can be a trap.

This paper introduces a new way to teach AI to solve these "Audio-Visual" mysteries without getting confused. The authors call their method SFFL (Separate First, Fuse Later).

Here is the breakdown of the problem and their solution, using simple analogies:

The Problem: The "Noisy Room" Effect

Current AI models are like a detective sitting in a very noisy room, trying to listen to two people talking at once.

The Visual Detective: Sees a sheep in the video.
The Audio Detective: Hears a dog barking.
The Mistake: Because the AI tries to process both at the exact same time, the visual clue (the sheep) gets in the way of the audio clue (the bark). The AI might get confused and say, "I hear a sheep!" even though the video clearly shows a dog barking and the sheep is silent. This is called cross-modal interference or hallucination. The AI is "hallucinating" a sound just because it saw an animal that usually makes that sound.

The Solution: The "Two-Step Interview"

The authors realized that instead of letting the AI mix everything together immediately, it should act like a smart detective who interviews witnesses separately before combining the stories.

1. Separate First (The Solo Interviews)
Instead of looking at the video and listening to the audio simultaneously, the AI is forced to do two separate "interviews":

Interview A: The AI looks only at the video and writes down what it sees (e.g., "I see a dog and a sheep"). It is not allowed to listen to the audio yet.
Interview B: The AI listens only to the audio and writes down what it hears (e.g., "I hear barking"). It is not allowed to look at the video yet.

To make sure the AI doesn't accidentally "peek" at the other witness during these interviews, the authors built a special digital wall called the Modality Asymmetric Attention Mask. Think of this as putting headphones on the visual detective so they can't hear the audio, and blindfolding the audio detective so they can't see the video.

2. Fuse Later (The Team Huddle)
Only after both detectives have written their separate reports does the AI bring them together.

It reads the "Video Report" and the "Audio Report."
It compares them: "The video says there's a sheep, but the audio report says there's no sheep sound, only barking."
The Decision: It concludes that the sound must belong to the dog, and the sheep is just there silently.

The "Preferred Evidence" Coach

The paper also mentions a clever training trick. The AI is taught to recognize which witness is the "star" for a specific question.

If the question is "What animal is making noise?", the AI learns that the Audio witness is the most important one.
If the question is "What color is the car?", the Visual witness is the star.

The AI is rewarded for correctly identifying which witness to trust most for each specific puzzle. This prevents it from blindly trusting the video just because it's usually the loudest voice in the room.

The Results

When the researchers tested this "Separate First, Fuse Later" method:

Fewer Mistakes: The AI stopped making up sounds for silent animals.
Better Accuracy: It got the right answers more often on standard tests.
Robustness: It became much harder to trick the AI with confusing videos or sounds.

In short: The paper teaches AI to stop trying to multitask too early. By forcing the AI to think about what it sees and what it hears separately first, and then combining those thoughts carefully at the end, it stops making silly mistakes and becomes a much better detective.

Technical Summary: Separate First, Fuse Later (SFFL)

1. Problem Statement

Current Audio-Visual Large Language Models (AVLMs) often suffer from cross-modal interference, where information from one modality (e.g., visual) improperly biases the interpretation of another (e.g., audio), leading to hallucinations. For instance, a model might attribute a sound to an animal visible in the video even if the audio contains no evidence of that sound. The authors attribute this to two primary factors:

Uncontrolled Joint Reasoning: Most existing AVLMs concatenate audio and visual tokens into a single stream, allowing irrelevant signals from one modality to interfere with the reasoning process of the other.
Modality Bias: Current models exhibit a visual-dominant bias, often relying heavily on visual cues while underutilizing or misinterpreting auditory evidence.

2. Methodology: Separate First, Fuse Later (SFFL)

The authors propose SFFL, a structured reasoning framework designed to isolate modality-specific processing before integrating evidence. The framework consists of three core components:

A. Preferred Evidence Modality (PEM) Data Pipeline

To train the model to identify which modality provides the decisive evidence for a specific question, the authors constructed the AVQA-PEM-14K dataset.

Process: For each AVQA instance, they generate Chain-of-Thought (CoT) answers under three input settings: Audio-only (A), Video-only (V), and Audio-Visual (AV).
Labeling: Based on consistency and correctness rates across these settings, instances are labeled as:
- PEM = Audio: Solvable by A and AV, but not V.
- PEM = Visual: Solvable by V and AV, but not A.
- PEM = Audio-Visual: Solvable only by AV (requiring both).
Usage: These labels serve as an auxiliary signal to encourage instance-dependent modality preferences.

B. Structured Reasoning with Modality Asymmetric Attention Mask (MAAM)

SFFL enforces a "Separate-then-Fuse" reasoning paradigm using specific control tags (<mod>, <v>, <a>, <sum>, <ans>) to structure the output.

Separation: The model generates distinct reasoning traces for visual (<v>) and audio (<a>) inputs independently.
Fusion: Evidence is only integrated at the <sum> stage before the final answer.
MAAM: To prevent information leakage during the separated reasoning stages (where standard causal attention might allow a token to attend to the wrong modality's input), the authors introduce a Modality Asymmetric Attention Mask. This mask explicitly blocks:
- Visual reasoning tokens from attending to audio input tokens.
- Audio reasoning tokens from attending to video input tokens or the visual reasoning span.

C. Two-Stage Reinforcement Learning (GRPO)

The model is trained using Group Relative Policy Optimization (GRPO) with two distinct stages and reward functions:

Stage 1 (Structure & Preference): Optimizes for the Modality Preference & Structure Reward ( $R_{mps}$ ). The model is rewarded for correctly identifying the PEM and adhering to the separated reasoning structure.
Stage 2 (Accuracy & Structure): Optimizes for a weighted sum of the Answer Accuracy Reward ( $R_{acc}$ ) and the structure reward. This stage refines the model's ability to produce correct answers while maintaining the isolated reasoning structure.

3. Key Contributions

SFFL Framework: A novel reasoning paradigm that enforces modality isolation during the Chain-of-Thought stage and delays fusion until the evidence integration stage, preventing premature cross-modal interference.
PEM Data Pipeline: A method to automatically construct training data with "Preferred Evidence Modality" labels, enabling the model to learn instance-dependent reliance on specific modalities.
Modality Asymmetric Attention Mask (MAAM): A mechanism to strictly enforce modality isolation during token generation, addressing the structural entanglement of audio-visual information in standard attention mechanisms.
Two-Stage GRPO Training: A training strategy that first aligns the model's evidence preference and reasoning structure, then optimizes for answer accuracy, balancing robustness and performance.

4. Experimental Results

The authors evaluated SFFL on AVHBench (cross-modal hallucination) and three general AVQA benchmarks (AVQA, Valor32k-AVQA v2.0, MUSIC-AVQA).

Hallucination Mitigation: On the AVHBench cross-modal hallucination benchmark, SFFL achieved an average relative gain of 11.17% compared to baselines.
General Performance: On general AVQA benchmarks, SFFL yielded an average relative gain of 5.16%.
Specific Gains: Using the Qwen3-Omni-30B backbone, SFFL improved the average hallucination score to 81.29 (vs. 73.12 zero-shot) and the general QA average to 80.24.
Ablation Studies:
- SFR (Separate-then-Fuse Reasoning): Primarily improved performance on tasks requiring consistency between modalities (MIS), reducing interference.
- MAAM: Provided consistent and significant gains across all metrics by effectively suppressing cross-modal leakage.
- Training Strategy: The two-stage GRPO approach outperformed both SFT and single-reward GRPO baselines, demonstrating the necessity of aligning modality preference before optimizing for answer accuracy.

5. Significance and Claims

The paper claims that SFFL offers an effective solution to cross-modal interference by fundamentally restructuring how AVLMs process information. Rather than relying on heuristic constraints or expensive post-hoc corrections, SFFL addresses the structural entanglement of audio and visual data at the source.

The authors emphasize that their method does not inject new knowledge but rather unlocks the latent capability of existing backbones. By aligning the model's reasoning process with the instance-specific evidence requirements (via PEM) and preventing information leakage (via MAAM), the framework significantly enhances both the accuracy and robustness of audio-visual reasoning, particularly in scenarios prone to hallucinations.