Technical Summary: Omni-Customizer

Problem Statement

The landscape of joint audio-video generation has been transformed by foundation models like Ovi and LTX-2. However, achieving cohesive multimodal customization—specifically the simultaneous preservation of visual identities and vocal timbres across multiple interacting subjects—remains a significant, underexplored challenge. Existing approaches face three primary limitations:

1. Architectural Incompatibility: Unimodal video customization and cross-modal driving pipelines require non-trivial architectural changes (e.g., separate audio towers) to function in joint generation, often falling outside their original design scope.
2. Weak Identity Binding: Unified models like DreamID-Omni utilize mechanisms (e.g., Syn-RoPE) that fail to achieve robust cross-modal identity binding, leaving identity cues vulnerable to positional decay and causing confusion in multi-subject scenarios.
3. Inherent Bottlenecks & Anomalies: Current frameworks built on open-source backbones suffer from limited speech reconstruction capacity in audio VAEs and unbalanced multilingual phonetic granularity. Crucially, they are prone to a "Caption Vocalization" anomaly, where the audio tower erroneously synthesizes non-speech descriptive captions into spoken audio.

Methodology

Omni-Customizer is an end-to-end framework built upon a symmetric Dual-Stream Diffusion Transformer (DiT) architecture, initialized from the pre-trained Ovi backbone. It addresses the aforementioned challenges through four core innovations:

1. Structured Omni-Captioning

The framework utilizes Multi-Modal Large Language Models (MLLMs) to re-caption data into a standardized format. This prompt structure explicitly links visual identity (FaceID) and vocal identity (SpeakerID) using anchor tokens (e.g., <sub1>). The prompt includes:

• Subject Descriptors: Visual and acoustic descriptions for each subject.
• Global Context: Environment and action descriptions.
• Speech Content: Dialogue strictly enclosed by <S> and <E> tags, ensuring clear demarcation between narrative text and spoken utterances.

2. Core Architectural Modules

• Omni-Context Fusion (OCF): Instead of relying on the diffusion backbone to resolve complex alignments, OCF elevates the text encoder into an active cross-modal alignment engine. It concatenates text embeddings, visual reference tokens, audio reference tokens, and TTS phoneme embeddings into a unified sequence. Through iterative transformer processing with residual connections, it enriches text representations with dense multimodal cues, facilitating precise identity binding.
• Semantic-Anchored Multimodal RoPE (SA-MRoPE): To resolve the structural mismatch between 1D text, 2D image, and temporal audio tokens, SA-MRoPE anchors reference tokens to their corresponding semantic subject descriptions within a unified 3D positional space. This ensures that visual and audio references are rigidly bound to their specific textual anchors (e.g., <sub1>), preventing identity entanglement in multi-subject scenes.
• Masked TTS Cross-Attention (MTP-CA): To eliminate the "Caption Vocalization" anomaly, MTP-CA employs a hard-gating mechanism. It injects phoneme priors strictly into text tokens located within the <S>...<E> span, ensuring that non-speech narrative regions receive zero phoneme-level excitation. This confines pronunciation enhancement exclusively to intended dialogue spans.

3. Training Strategy

The authors devise a comprehensive training strategy to handle skewed language distributions (e.g., predominantly Chinese data) and prevent the degradation of foundational priors:

• Interleaved Modality-Decoupled Training: The training alternates between Joint Audio-Video Generation (JAVG) steps and TTS-only steps. The TTS-only steps expand the audio batch size and freeze cross-modal pathways, acting as a regularizer that allows the audio branch to rapidly acquire multilingual capabilities without compromising lip-sync or cross-modal alignment priors.
• Progressive In-Pair to Cross-Pair Curriculum: Training begins with single-subject alignment (in-pair data) to establish basic customization capabilities. It then escalates to multi-subject disentanglement (cross-pair data), where references are decoupled from target generation to force the learning of high-level, robust identity features.

Key Contributions

1. Omni-Customizer Framework: An end-to-end system for human-centric joint audio-video customization that achieves precise multimodal identity binding.
2. Novel Modules: The introduction of OCF for dense cue enrichment, SA-MRoPE for structured semantic anchoring, and MTP-CA for preventing speech leakage.
3. Advanced Training Strategy: An interleaved training approach and progressive curriculum that enable rapid multilingual adaptation and robust identity disentanglement without sacrificing alignment.
4. OC-Bench: A comprehensive benchmark comprising 300 test cases across single-subject, two-person dialogue, and complex multi-subject scenarios, accompanied by a diverse multi-subject multimodal dataset.

Results

Extensive experiments on OC-Bench demonstrate that Omni-Customizer achieves State-of-the-Art (SOTA) performance across multiple metrics:

• Identity Preservation: Outperforms baselines in Face Similarity (0.812) and Timbre Similarity (0.514), significantly reducing identity confusion in multi-subject interactions.
• Synchronization: Achieves superior audio-video synchronization (Sync-C/D) and the lowest Word Error Rate (WER), indicating high speech fidelity.
• Quality: Maintains high aesthetic and imaging quality in video, and high perceptual quality in audio.
• Ablation Studies: Confirm that removing any proposed module (OCF, SA-MRoPE, MTP-CA) or strategy (Curriculum, Interleaved training) leads to significant performance drops, such as identity drift, simultaneous speaking errors, or the re-emergence of caption vocalization.

Significance

The paper claims that Omni-Customizer bridges a critical gap in generative AI by enabling robust, high-fidelity joint audio-video customization in complex, human-centric scenarios. By solving the problems of identity entanglement and speech leakage, the framework enables realistic multi-subject conversational generation that was previously unattainable with open-source models. The work establishes a new paradigm for multimodal customization, demonstrating that structured semantic anchoring and decoupled training strategies are essential for scaling joint generation to interactive, multi-character applications.

Note: The authors acknowledge current limitations, noting that generations are currently bounded to 720P resolution and 10-second durations, with future work focused on scaling to higher resolutions and longer sequences.