Technical Summary: OmniMate

Problem Statement

Recent advances in diffusion-based generative models have enabled high-fidelity, real-time audio-driven avatar generation and unified audio-visual synthesis. However, extending these capabilities to open-ended, real-time interactive streaming presents two critical challenges:

1. Unknown Generation Horizon: Unlike conventional generation with predefined speech segments, interactive streaming requires the model to generate responses based on execution prompts without knowing in advance when the content is complete. Fixed durations risk truncating responses or generating redundancy, while explicit completion detection introduces latency that hinders real-time interaction.
2. Long-Term Cross-Modal Identity Drift: In streaming scenarios with limited temporal context, visual identity cues (appearance, pose) and speaker identity cues (voice timbre) tend to fade over time. Single-reference approaches struggle to maintain consistency across unseen poses, viewpoints, and multiple conversational turns, leading to identity drift.

Methodology

The authors propose OmniMate, a unified framework designed for open-ended, real-time interactive audio-visual avatar generation. The system jointly synthesizes speech, facial motions, body gestures, and action-related sound effects while maintaining seamless transitions between "listening" and "execution" states. The framework is built upon the LTX2.3 foundation model and incorporates two novel modules:

1. Generation Progress Controller (GPC)

The GPC addresses the challenge of adaptive response termination and seamless state transitions.

• Mechanism: It explicitly models the generation progress of each streaming chunk. Given an estimated execution duration from an LLM, the GPC maps latent-aligned progress to continuous time embeddings (ranging from 1 to 1000 for execution, and 0 for listening).
• Injection: These progress signals are injected into the latent representations via a Multi-Layer Perceptron (MLP) and Sinusoidal Positional Encoding.
• Training Strategy: The authors construct state-transition training data using a DMD-distilled bidirectional teacher model. This teacher generates sequences with explicit transitions between execution and listening states. A segment-wise text cross-attention mask is applied to ensure the student model learns to attend to specific prompts (execution vs. listening) based on the injected progress signal, preventing semantic leakage across state boundaries.

2. Multi-Reference Conditioning Module (MRCM)

The MRCM addresses long-term identity preservation by leveraging persistent cues.

• Visual Conditioning: Instead of a single reference, the module concatenates multiple reference images (up to four) with the noisy video latents along the temporal dimension. Negative Rotary Positional Embeddings (RoPE) are applied to these reference tokens to prevent them from interfering with the temporal positions of generated frames.
• Speaker Conditioning: A training-free strategy is employed where a reference audio segment (or a model-generated warm-up utterance) is used to replace generated audio latents during the denoising process. This ensures consistent voice timbre and speaker characteristics throughout the interaction.

Training and Inference Pipeline

The training process involves four stages:

1. Fine-tuning LTX2.3 with visual identity conditioning.
2. Distilling a few-step bidirectional teacher model using Distribution Matching Distillation (DMD) and constructing state-transition ODE trajectories.
3. ODE Initialization: Jointly optimizing the GPC and the causal student model.
4. Self-Forcing DMD: Further distilling the model to enforce autoregressive behavior, where each chunk attends only to reference chunks, the first chunk, and adjacent chunks.

During inference, the system uses an LLM to generate response content and estimated duration. The GPC converts this duration into progress signals to guide the streaming generation, while the MRCM maintains identity consistency.

Key Contributions

1. OmniMate Framework: An open-ended, real-time interactive audio-visual avatar generation framework capable of jointly synthesizing speech, expressions, gestures, and environmental sounds with low latency.
2. Generation Progress Controller (GPC): A novel controller that explicitly models generation progress, enabling the model to follow desired execution schedules, achieve seamless transitions between listening and execution states, and respond to new instructions with minimal latency.
3. Multi-Reference Conditioning Module (MRCM): A module that utilizes multiple reference images and a reference audio segment to preserve long-term visual and speaker identity consistency during extended streaming interactions.

Experimental Results

The authors evaluated OmniMate on an interaction-oriented adaptation of VerseBench, comparing it against state-of-the-art offline (e.g., Ovi-1.1, MOVA, UniVerse-1) and real-time (e.g., OmniForcing, Hallo-Live) models.

• Performance Metrics: OmniMate achieved the highest generation speed (27.64 FPS) and the lowest Time-to-First-Frame (3.49s) among real-time methods, while maintaining competitive visual quality and lip-sync accuracy.
• Consistency: The model demonstrated superior audio-visual consistency (IB-Score) and strong preservation of audio identity (A-ID) and visual identity (V-ID) over long durations.
• Ablation Studies: Experiments confirmed that removing the GPC led to inaccurate state transitions and repetitive speech, while removing MRCM components resulted in degraded visual identity under varying viewpoints and inconsistent speaker characteristics.
• Long-Duration Stability: Evaluations at 30s, 60s, and 240s showed no significant degradation in quality or consistency, validating the framework's capability for extended interactions.

Significance and Claims

The paper claims that OmniMate provides a robust solution for real-time interactive avatar systems by solving the fundamental trade-offs between generation horizon uncertainty and long-term identity preservation. By integrating adaptive progress control and multi-reference conditioning, the framework enables natural, immersive, and coherent multi-turn conversations where the avatar can seamlessly switch between listening and speaking without identity drift or latency-induced awkwardness. The authors position OmniMate as a step toward realistic, responsive, and user-avatar conversational behaviors that maintain high fidelity over extended periods.

Limitations

The authors acknowledge that extremely long responses or inaccurate duration estimation may occasionally lead to word omissions or repetitions. Additionally, current identity preservation mechanisms limit the generation of actions involving large appearance or scene changes. Future work is proposed to focus on improved progress modeling and extending the framework toward autonomous long-horizon interactions.