Technical Summary: Beyond Time Shifts: Adapting Omni-LLM as a Reference-Free Evaluator for Generative Audio-Visual Models

1. Problem Statement

As audio-visual (AV) generative models evolve into "world simulators," evaluating the consistency of cross-modal dynamics has become critical. However, existing evaluation metrics face a fundamental mismatch with modern generative outputs:

• Structural Assumption Failure: Traditional metrics (e.g., onset-based synchronization, cross-modal contrastive similarity) presume that audio and video are structurally correct and merely require temporal alignment. They fail to detect structural hallucinations (e.g., a plastic cup falling producing a glass-shattering sound), asymmetric relations (visual actions without corresponding audio), and temporally diffuse events (continuous pouring where pitch fails to match fluid level).
• The Annotation Paradox: Assessing these complex errors requires human cognition. However, human evaluators naturally rely on relative comparisons (judging which of two flawed outputs is better), whereas deployable automated metrics must be reference-free, outputting an absolute scalar score for a single generated pair.
• Limitations of Current AI Judges: Existing "LLM-as-Judge" approaches often rely on discrete text categories or unimodal fidelity measures, which introduce quantization artifacts and fail to capture the fine-grained, causal-semantic consistency required for AV synchronization.

2. Methodology

The authors propose a three-stage framework to resolve the paradox between relative human perception and absolute automated scoring, transforming an Omni-LLM into a continuous, reference-free evaluator.

2.1 SynthSync Dataset

To ground the evaluation in authentic generative failures rather than artificial time shifts, the authors constructed SynthSync:

• Construction: Real-world video clips are processed by 10 state-of-the-art Video-to-Audio (V2A) foundation models to generate multiple distinct acoustic tracks per video.
• Annotation: Expert annotators perform pairwise comparisons ($a_i$ vs. $a_j$) against a fixed visual anchor to determine which audio better matches the video.
• Outcome: This yields a robust, global ordinal ranking based on empirical win-rates, capturing complex failure modes like semantic hallucinations and causal omissions.

2.2 Continuous Omni-LLM Architecture

The authors adapt an Omni-LLM (specifically Qwen2.5-Omni-3B) to bypass discrete token decoding:

• Latent Projection: The discrete language head is replaced with a continuous projection head (MLP) that maps the model's hidden representation of a special [SCORE] token directly to a scalar value $s \in \mathbb{R}$.
• Objective: This design strips away language-generation stochasticity, creating a fully differentiable mapping optimized for relative continuous scaling.

2.3 Two-Stage Training Paradigm

The training process aligns the continuous estimator with human preferences through two distinct stages:

Stage I: Latent Preference Alignment (Curriculum Learning)

• Loss Function: The model is trained using the Bradley-Terry-Luce (BTL) formulation to minimize the loss over logit decision boundaries for pairwise preferences ($a_i \succ_v a_j$).
• Curriculum Pacing: To prevent gradient collapse due to the variance in error types, training begins with pairs having large rank discrepancies (e.g., best vs. worst) and gradually decays the threshold to expose the model to finer, microscopic comparisons.

Stage II: Causal Topologies via R-GRPO

• Problem: Pairwise learning ensures local margins but does not guarantee global transitivity or a coherent score distribution.
• Solution: The authors introduce Real-Valued Group Relative Policy Optimization (R-GRPO).
  • Stochastic Reparameterization: The deterministic scalar output is treated as the mean ($\mu_\theta$) of a Gaussian policy, injecting necessary uncertainty for policy gradient optimization.
  • Exploratory Rollouts: The model samples $N$ parallel execution paths by perturbing scores within a designated variance $\sigma^2$.
  • Composite Reward: A holistic reward function evaluates the entire listwise distribution against the ground-truth topology, optimizing for structural attenuation, monotonic order, and extreme target bounding.
  • Objective: The policy is updated using a clipped surrogate loss with a KL divergence penalty to maintain proximity to the reference backbone.

3. Key Contributions

1. SynthSync: The first dataset dedicated to authentic generative synchronization failures, moving beyond artificial time-shift benchmarks to capture semantic and causal errors.
2. Continuous Omni-LLM Evaluator: A novel architecture that repurposes multimodal foundation models for continuous, reference-free causal consistency scoring, avoiding the quantization biases of discrete text outputs.
3. R-GRPO Framework: A scalable reinforcement learning algorithm that aligns continuous latent policies with discrete human preferences, establishing a new standard for physically grounded generative evaluation.
4. SyncBench & SOTA Alignment: The establishment of a standardized benchmark (SyncBench) and the demonstration of state-of-the-art alignment with human preference in causal evaluation.

4. Experimental Results

• Benchmark Performance: On the SynthSync benchmark, the proposed method achieves state-of-the-art performance in human preference alignment. It outperforms off-the-shelf Omni-LLMs, traditional non-LLM metrics (e.g., AV-Align, DeSync), and other trained LLM evaluators.
  • Key Metrics: The full model (with R-GRPO) achieves 72.38% Pairwise Accuracy, 61.40% Top-1 Accuracy, and a Kendall's $\tau$ of 0.4899, significantly surpassing baselines.
• SyncBench Evaluation: When applied to evaluate six cutting-edge AV-Gen models (including Sora-2, Veo-3.1, and LTX-2), the metric successfully stratifies performance, identifying industry leaders while traditional metrics produce volatile and contradictory rankings.
• Test-Time Scaling: When deployed as a reward signal for Best-of-N (BoN) sampling, the metric demonstrates superior cross-metric generalization, outperforming random selection and other reward signals in improving model quality across independent evaluation metrics.
• Ablation Studies:
  • Representation: Continuous scalar ranking significantly outperforms scalar regression and text-based decoding (CE/KL), confirming that avoiding discrete tokens preserves relative topological order.
  • Components: Incremental ablation shows that both the SynthSync dataset and the R-GRPO stage are critical, with R-GRPO providing the largest gains in rank correlation.
  • Reward Formulation: A "Dual Reward" strategy (combining pairwise and listwise constraints) yields optimal performance, balancing global structural consistency with local comparative accuracy.
  • Exploration Variance: An intermediate exploration variance ($\sigma=2.5$) in the Gaussian policy provides the best balance between exploring global topologies and maintaining policy stability.

5. Significance and Claims

The paper claims to resolve a critical bottleneck in AV generation: the lack of metrics capable of understanding intricate physical interactions between sight and sound. By shifting the evaluation paradigm from low-level signal correlation to visually grounded causality, the proposed framework:

• Bridges the Human-Machine Gap: It successfully distills complex, comparative human judgment into a continuous, reference-free, and globally consistent automated metric.
• Enables Reliable Benchmarking: It provides the field with a standardized tool (SyncBench) to assess the physical grounding of nascent world simulators, distinguishing between models that merely align signals and those that understand causal dynamics.
• Offers Dual Utility: The estimator serves not only as a reliable evaluator but also as an actionable reward signal for training and test-time scaling, advancing the development of more physically consistent generative models.