Technical Summary: Boosting Omni-Modal Language Models via Staged Post-Training with Visually Debiased Evaluation

Problem Statement

Omni-modal language models (LLMs) are designed to jointly process audio, visual, and textual inputs. However, the paper argues that strong performance on existing benchmarks does not necessarily indicate genuine omni-modal integration. A significant portion of benchmark gains may stem from "visual shortcuts," where models solve queries using visual evidence and the question alone, bypassing the need for audio processing. This modality leakage causes evaluations to overestimate a model's true omni-modal capabilities. Furthermore, it remains an open question whether strengthening individual bi-modal competencies (vision-language and audio-language) is sufficient for omni-modal understanding, or if explicit omni-modal data and optimization signals are required.

Methodology

1. OmniClean: A Visually Debiased Evaluation Protocol

To address evaluation leakage, the authors introduce OmniClean, a protocol that audits nine existing omni-modal benchmarks (e.g., Daily-Omni, WorldSense, OmniBench) to filter out visually solvable queries.

• Visual-Only Probing: For each query, the audio input is withheld, and a strong vision-language model (Qwen3-VL-30B-A3B-Thinking) is prompted to answer using only the visual input (frames sampled at 2 fps) and the text question.
• Filtering Rule: If a query can be answered correctly by the visual-only model in at least one of 16 rollouts (pass@16), it is marked as "visually answerable" and excluded from the cleaned evaluation view.
• Exceptions: Two benchmarks, AV-Odyssey and CG-AV-Counting, are retained in full. AV-Odyssey is excluded from filtering because its answer options sometimes contain audio inputs that pure vision-language models cannot process. CG-AV-Counting is retained to maintain evaluation stability, as further filtering would drastically reduce the query count.
• Result: The protocol audits 16,968 queries, retaining 8,551 queries that constitute the OmniClean dataset. This dataset serves as a "visually debiased" testbed where performance is less likely to be driven by visual shortcuts.

2. OmniBoost: Staged Post-Training Recipe

Using OmniClean, the authors evaluate OmniBoost, a three-stage post-training pipeline based on the Qwen2.5-Omni-3B model. The study investigates whether bi-modal supervision alone is sufficient or if explicit omni-modal signals are necessary.

• Stage 1: Mixed Bi-modal SFT (Control):

  • Goal: Test if balanced supervision across bi-modal pairs transfers to omni-modal tasks.
  • Data: A balanced mixture (1B output tokens each) of audio-text, image-text, video-text, and pure-text data. Video data is augmented with dense captions and reasoning traces.
  • Training: 1 epoch of supervised fine-tuning (SFT) with modality-agnostic packing.

• Stage 2: Mixed-Modality RLVR:

  • Goal: Optimize reasoning behavior using explicit omni-modal signals.
  • Data: A curated mixture including text-only, image-text, video-text, audio-image-text, and audio-video-text queries. The mixture is heavily weighted toward audio-video-text (54.8%).
  • Algorithm: Direct Preference Optimization (DAPO) with verifiable rewards. The reward schedule shifts from format accuracy to grounded answer accuracy after 500 steps.
  • Training: 1,200 steps of RLVR.

• Stage 3: Self-Distillation SFT:

  • Goal: Amplify reasoning patterns using synthetic data without a stronger external teacher.
  • Synthetic Query Construction: Seed videos (LLaVA-Video) are segmented and captioned by Step-Audio-R1 (audio) and Qwen3-VL (video). An entity-relation graph is constructed using gpt-oss-120b to scaffold spatial and temporal relations. These elements are combined to generate hard-matchable audio-visual-text questions.
  • Filtering: The 3B RLVR checkpoint generates 8 rollouts per synthetic query. Three progressive filtering passes (F1–F3) remove queries that are too easy/hard, contain perception errors, or have inconsistent reasoning/answers.
  • Training: SFT on the filtered synthetic traces, adjusting data ratios.

Key Results

Evaluation on OmniClean

The authors report results on the OmniClean dataset, comparing the three stages against open-source references (Qwen2.5-Omni-7B, Qwen3-Omni-30B).

• Stage 1 (Bi-modal SFT): Yields limited and uneven gains. The macro average improves only slightly (24.92 → 26.49), indicating that bi-modal competence alone does not guarantee consistent omni-modal performance.
• Stage 2 (RLVR): Delivers the first broad improvement. The macro average jumps to 31.43, outperforming the Stage 1 baseline significantly. This suggests that explicit omni-modal optimization signals are critical.
• Stage 3 (Self-Distillation): Shows profile-dependent gains. While the macro average slightly dips to 31.03 compared to Stage 2, the query-weighted average rises to 32.15. This is largely driven by performance gains on large retained subsets like AV-Odyssey.
• Comparison to Larger Models: After Stage 3, the 3B model becomes comparable to larger open-source references and slightly exceeds Qwen3-Omni-30B-A3B-Instruct under the query-weighted average, despite not distilling answers from a stronger omni-modal teacher.

Ablation Studies

• Data-Centric Ablation: Training the base Qwen2.5-Omni-3B directly on filtered synthetic datasets (F1, F2, F3) without the full RLVR stage still yields improvements over the baseline, confirming the utility of the synthetic supervision.
• Correlation Shifts: Cleaning the benchmarks reduces the correlation between omni-modal scores and unimodal (vision-only) reference strengths, particularly for benchmarks like WorldSense and Daily-Omni, validating that the evaluation is now more reflective of true omni-modal integration.

Key Contributions

1. OmniClean Dataset: A visually debiased evaluation view comprising 8,551 queries across nine benchmarks, constructed by filtering out queries solvable by visual-only probing.
2. OmniBoost Recipe: A staged post-training framework demonstrating that balanced bi-modal SFT is insufficient for omni-modal gains, whereas mixed-modality RLVR and self-distillation on synthetic omni-modal queries provide substantial improvements.
3. Synthetic Query Construction: A method for generating hard-matchable audio-visual-text queries using entity-relation scaffolds and captions, enabling effective self-distillation without a stronger teacher model.
4. Empirical Evidence: Demonstration that a 3B model, through careful staged post-training, can achieve performance comparable to or exceeding 30B parameter models on visually debiased benchmarks.

Significance and Claims

The paper claims that omni-modal progress is more meaningfully assessed when evaluation controls for visual leakage. The findings suggest that:

• Raw benchmark gains often reflect visual shortcut exploitation rather than genuine integration.
• Small omni-modal models can gain substantial capability through carefully staged post-training and self-distilled supervision, even without access to stronger teacher models.
• Explicit omni-modal data and optimization signals (specifically RLVR) are necessary to bridge the gap between bi-modal competence and true omni-modal understanding.

The authors release the OmniClean evaluation data to support future leakage-aware omni-modal evaluation. The study is scoped to the Qwen2.5-Omni-3B lineage and the specific visual-only probing protocol described.