💻 computer science

MultiRef-Compass: Towards Comprehensive Evaluation of Multi-Reference-to-Audio-Video Generation

This paper introduces MultiRef-Compass, a unified benchmark comprising 350 curated samples and a four-dimensional evaluation protocol with 14 sub-metrics, designed to comprehensively assess the emerging capability of multi-reference-to-audio-video generation systems in preserving, binding, and composing multiple references into coherent synchronized content.

Original authors: Xiaohan Zhang, Yuqing Wen, Junlin Chen, Yuqi Tang, Yiting He, Lizhuo Shao, Weiming Zhu, Tengfei Liu, Yang Shi, Jialu Chen, Yuanxing Zhang, Huaxiong Li

Published 2026-07-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xiaohan Zhang, Yuqing Wen, Junlin Chen, Yuqi Tang, Yiting He, Lizhuo Shao, Weiming Zhu, Tengfei Liu, Yang Shi, Jialu Chen, Yuanxing Zhang, Huaxiong Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a director trying to film a movie scene. In the old days of AI video, you could only give the computer a simple script: "Make a video of a cat." The AI would guess what the cat looked like, where it was, and what sound it made. But now, creators want more control. They want to say, "Use this specific photo of my dog for the face, this other photo for his side profile, and this third photo for his favorite toy, all while he barks at a specific sound." This is the world of Multi-Reference-to-Audio-Video (MR2AV) generation. It's like asking an AI to be a master chef who must not only cook a meal but also use specific ingredients from three different jars, arrange them on the plate exactly as shown in a picture, and make sure the sizzling sound matches the cooking action perfectly. The problem is, we didn't have a good way to grade how well the AI was doing this complex dance. We had tests for simple recipes, but nothing for this multi-ingredient, multi-sensory challenge.

Enter MultiRef-Compass, a new "report card" designed by researchers to test exactly this skill. Think of it as a rigorous taste-test for AI video makers. The team built a special dataset of 350 carefully crafted scenarios—like a cooking show challenge where every contestant gets the exact same three photos of a person, a shoe, and a diner, plus a script saying, "The person tries on the shoe and asks if it fits." They then asked eight different AI models (including big names like Kling, Seedance, and Gemini) to generate the video.

The researchers didn't just look at whether the video looked "nice." They broke the grading down into four specific categories, like a detailed rubric:

Basic Quality: Does the video look clear and sound good, or is it blurry and crackly?
Reference Consistency: Did the AI actually use the photos you gave it? Did it keep the person's face the same, or did it accidentally swap them with a stranger? Did it glue the shoe onto the foot correctly, or did it just paste a sticker of a shoe on the screen?
Audio-Visual Consistency: When the person in the video speaks, do their lips move? When they step on the floor, do you hear a footstep?
Instruction Following: Did the AI do exactly what the script asked, or did it ignore the part about the shoe and just show the person walking?

To grade these, they used a mix of computer tools and a "super-smart AI judge" (a Large Multimodal Model) that acts like a film critic. They even added a special "re-judge" step to make sure the AI critic wasn't being too harsh or too easy, ensuring the scores were fair.

The results were a reality check. While some models are getting really good at making things look pretty, they are still struggling with the hard stuff. The paper suggests that current models often fail at binding—they might recognize the face in the photo but forget to attach it to the body, or they might mix up which person is supposed to be speaking. One model, Seedance 2.0, came out on top overall because it was the most balanced, doing well in all four categories. However, even the best models showed "substantial room for improvement." They found that when you ask an AI to juggle multiple references and complex instructions at once, it often drops the ball, creating videos where the characters look like cutouts, the voices don't match the speakers, or the story gets shuffled.

In short, MultiRef-Compass shows us that while AI video is getting impressive, it's not quite ready to be a reliable director for complex, multi-reference stories yet. It's a tool that needs more training to understand how to weave multiple clues together into a single, coherent, and natural-looking scene. The benchmark itself is now open for everyone to use, acting as a foundation to help future AI models learn how to be better at this specific, tricky kind of creative work.

Technical Summary: MultiRef-Compass

Problem Statement

The field of audio-video generation has evolved from text-driven (T2AV) and single-reference (R2AV) paradigms to Multi-Reference-to-Audio-Video (MR2AV) generation. In MR2AV, models must synthesize coherent audio-video content conditioned on multiple references (images, videos, audio) and textual instructions. While existing benchmarks like T2AV-Compass, UniVBench, and OpenS2V-Nexus evaluate instruction following, subject consistency, or audio-visual alignment, they largely fail to address the specific complexities of the MR2AV setting.

Current evaluation protocols are insufficient because they do not rigorously test:

Cross-reference understanding: The ability to distinguish whether multiple reference images depict different views of the same subject or distinct entities.
Compositional binding: The ability to correctly assign roles, actions, and interactions to specific referenced entities across visual and audio modalities.
Natural visual integration: The ability to blend referenced appearances into a generated scene without producing "copy-and-paste" artifacts or rigid identity splitting.

Consequently, there is a lack of a unified benchmark capable of diagnosing whether models can jointly interpret, bind, and compose multiple references into synchronized audio-visual events.

Methodology

1. Dataset Construction: MultiRef-Compass

The authors introduce MultiRef-Compass, a benchmark comprising 350 carefully curated samples constructed via a scalable, taxonomy-driven asset-composition pipeline. The dataset is built in three stages:

Asset-Pack Curation: Collection of royalty-free scene, video, and voice packs, alongside subject packs constructed from real-world videos or AIGC-generated multi-view identities. All assets are manually verified for identity consistency and quality.
Board-Specific Asset Composition: Samples are organized into four "boards" to control complexity:
- Board 1 (B1): Same-identity multi-view conditioning (testing view consistency).
- Board 2 (B2): Heterogeneous compositions (subject, object, scene).
- Board 3 (B3): Multi-entity binding (testing role assignment and interaction).
- Board 4 (B4): Extended challenge with video and audio references.
Structured Prompt Generation: Prompts are generated using a structured schema (via GPT-5.5) rather than free-form text, ensuring precise specification of visual, audio, speech, and temporal requirements.

2. Evaluation Framework

MultiRef-Compass employs a hybrid evaluation framework integrating automatic metrics with a rejudging-enhanced MLLM-as-a-Judge protocol. The evaluation covers four dimensions with 14 sub-metrics:

Basic Quality (BQ): Assesses technical fidelity.
- Visual Technical Quality (VTQ): DOVER++ scores.
- Audio Technical Quality (ATQ): Audiobox scores.
- Anatomical Quality (AQ): MLLM checklist for biological plausibility and 3D coherence.
Reference Consistency (RC): Assesses preservation and binding.
- Entity Fidelity (EF): Combines embedding similarity (SigLIP/YOLO-World) with a paste-naturalness coefficient ( $P_{paste}$ ) estimated by an MLLM to penalize "sticker-like" copy-paste artifacts.
- Detail Preservation (DP): MLLM evaluation of fine-grained details.
- Binding Correctness (BC): MLLM detection of identity swaps or attribute leakage.
Audio-Visual Consistency (AVC): Assesses synchronization.
- Speech-Lip Synchronization (SLS): LatentSync on MLLM-filtered stable frontal faces.
- Event-Sound Matching (ESM) & Source Correctness (SC): MLLM evaluation of sound-event alignment and speaker attribution.
- Timbre Similarity (TS): Speaker embedding similarity for voice references.
Instruction Following (IF): Assesses adherence to prompts.
- Includes Visual Task Following (VT), Audio Task Following (AT), Speech Content Accuracy (SCA), and Temporal Order Following (TO), all evaluated via MLLM checklists.

Rejudging Mechanism: To mitigate MLLM inconsistency, a lightweight rejudging stage compares initial scores and rationales across models for the same sample, correcting overly strict or permissive judgments while preserving the original metric definitions.

Key Contributions

MultiRef-Compass Benchmark: The first unified benchmark for MR2AV generation, featuring 350 taxonomy-driven samples covering multi-view preservation, multi-entity binding, and human-object-scene composition.
Hybrid Evaluation Protocol: A framework that integrates automatic metrics with a rejudging-enhanced MLLM-as-a-Judge system, specifically designed to handle the nuance of multi-reference binding and cross-modal consistency.
Comprehensive Empirical Analysis: Extensive evaluation of eight representative MR2AV systems (6 closed-source, 2 open-source), revealing specific failure modes and capability gaps.

Experimental Results

The authors evaluated eight models (including Seedance 2.0, Kling 3.0, HappyHouse 1.1, Gemini-Omni, Wan 2.7, Vidu Q3-Mix, SkyReels-V3, and Wan2.1-VACE) across the four dimensions.

Performance Variance: No single model achieved consistently strong performance across all dimensions. Seedance 2.0 ranked first overall due to balanced capabilities, while Kling 3.0 excelled in Reference Consistency and Gemini-Omni in Basic Quality and Audio-Visual Consistency.
Reference Consistency Challenges: Models struggled significantly with complex compositions. Identity splitting (treating multiple views of one subject as distinct entities) and copy-and-paste artifacts were common. The proposed $P_{paste}$ metric successfully downgraded models that achieved high embedding similarity through rigid copying rather than natural synthesis.
Open-Source vs. Closed-Source: Open-source models (SkyReels-V3, Wan2.1-VACE) lagged significantly behind closed-source systems, particularly in Reference Consistency and Instruction Following, indicating difficulties in understanding relationships between multiple references.
Instruction Following: Models exhibited modality-specific strengths; for instance, Seedance 2.0 was strong in visual tasks, while Kling 3.0 performed better on audio and speech instructions.
Human Alignment: The benchmark scores showed strong Pearson correlations with human preferences (0.89–0.96) across all dimensions, validating the evaluation protocol.

Significance and Claims

The paper positions MultiRef-Compass as a foundational diagnostic tool for the emerging field of MR2AV generation. The authors claim that:

Current benchmarks are insufficient for the MR2AV setting, as they fail to distinguish between simple reference fidelity and the complex reasoning required to bind multiple references.
The proposed benchmark exposes specific failure modes (e.g., identity splitting, unnatural integration) that are obscured by aggregate quality scores in existing evaluations.
Future progress in MR2AV depends not just on improving individual components (visual or audio quality) but on the joint optimization of perceptual quality, structural plausibility, multi-reference coherence, and instruction adherence.
The benchmark provides a scalable, auditable, and interpretable testbed for evaluating future omni-reference audio-video generation systems.