Personalizing Causal Audio-Driven Facial Motion via Dynamic Multi-modal Retrieval

1. Problem Statement

The paper addresses the critical challenge of generating high-fidelity, personalized facial animations from live audio streams in real-time (low-latency) settings, such as AR/VR telepresence. Existing methods face two primary limitations:

• Latency vs. Causality: High-quality synthesis often requires "audio look-ahead" (buffering future audio frames) or computationally intensive iterative sampling (e.g., diffusion models), which introduces unacceptable latency for real-time streaming.
• Loss of Identity/Style: Deterministic mappings often suffer from "regression to the mean," producing averaged, over-smoothed motions that fail to capture unique individual traits (e.g., asymmetric smirks, specific blinking patterns). Current personalization methods often rely on static, pre-encoded embeddings or rigid calibration scripts, failing to adapt dynamically to unstructured reference data.

2. Methodology: The Fallingwater Framework

The authors propose Fallingwater, an end-to-end causal framework that combines a hierarchical motion codec with a dynamic multi-modal retrieval mechanism.

A. Causal Hierarchical Motion Codec

To enable streaming without look-ahead, the system uses a multi-scale temporal auto-encoder:

• Hierarchical Discrete Latent Space: Facial motion is decomposed into discrete tokens across multiple temporal resolutions ($L$ levels).
  • Coarse Levels: Capture long-term prosody and global dynamics (e.g., head tilts, brow elevations).
  • Fine Levels: Capture high-frequency details (e.g., lip articulation, subtle muscle movements).
• Binary Spherical Quantization (BSQ): Motion features are projected onto a hypersphere for robust quantization, creating a discrete codebook.
• Strict Causality:
  • The Encoder is non-causal (used offline for training) to capture rich temporal context.
  • The Decoder is strictly causal, using a triangular attention mask and causal nearest-neighbor interpolation (avoiding linear interpolation that requires future values). This allows for zero-lookahead decoding, synthesizing frames immediately upon audio arrival.

B. Streaming Generation Strategy

The generator is an autoregressive transformer that predicts motion tokens in an Interleaved Coarse-to-Fine manner:

• As audio arrives, the model predicts coarse "anchor" tokens first to establish global trends.
• It then interleaves the prediction of finer tokens to add detail.
• This asynchronous prediction minimizes latency while maintaining multi-scale expressivity.

C. Dynamic Multi-modal Style Retriever

To solve the under-determined nature of audio-to-motion mapping (where audio alone cannot resolve unique idiosyncrasies), the system introduces a dynamic retriever:

• Joint Query: The retriever queries an unstructured reference library using a combined embedding of the current audio and historical motion (the most recent generated frames).
• Dynamic Priors: Unlike static ID embeddings, this mechanism dynamically extracts stylistic priors (e.g., specific lip-pursing habits) from arbitrary video clips of the target identity.
• Re-querying Training Strategy: To prevent "exposure bias" (where the model fails when its own generated motion deviates from ground truth), the training process includes a second pass. The model re-queries the style library using its own noisy generated motion as the query, forcing it to learn to "recover" correct stylistic anchors even from imperfect inputs.

3. Key Contributions

1. Causal Hierarchical Architecture: A novel auto-encoding architecture that enables zero-lookahead streaming synthesis by decoupling high-frequency articulation from low-frequency expression while maintaining strict causality.
2. Multi-modal Style Retrieval: A mechanism that dynamically queries unstructured reference libraries using joint audio-motion embeddings to extract identity-specific priors, eliminating the need for rigid calibration or static templates.
3. Unified Framework: The integration of these components achieves state-of-the-art (SOTA) performance in lip-sync accuracy and identity consistency under strict real-time constraints.

4. Experimental Results

The system was evaluated on a dataset of 832 subjects (469.6 hours of video) and compared against SOTA methods (DiffPoseTalk, ARTalk, MemoryTalker, etc.).

• Quantitative Metrics:
  • Sync Score: 0.509 (SOTA), significantly outperforming baselines (e.g., 0.479 for ARTalk).
  • Similarity Score (ID Consistency): 0.718 (SOTA), demonstrating superior preservation of unique identity traits compared to 0.596 (ARTalk) and 0.117 (MemoryTalker).
  • ID-FED/FPD: Lowest Fréchet distances, indicating the generated motions are statistically closest to the ground truth identity distributions.
• User Study: In a study with 15 participants (450 comparisons), Fallingwater was preferred over all baselines in Lip Sync (up to 86.9% preference), ID Similarity, and Visual Naturalness.
• Ablation Studies:
  • Removing the hierarchical codec caused a significant drop in quality.
  • Removing the multi-modal retriever or using only audio/motion queries drastically reduced ID similarity.
  • The Re-querying strategy was proven essential to prevent inference-time drift and collapse into trivial solutions.

5. Significance

Fallingwater represents a major leap forward in immersive telepresence. By successfully reconciling ultra-low latency (strict causality) with high-fidelity personalization, it solves a long-standing bottleneck in digital avatar technology.

• Practical Impact: It allows for real-time interaction where avatars can mimic a user's unique speaking style and mannerisms without pre-processing or rigid enrollment, simply by referencing unstructured video data.
• Technical Innovation: The combination of causal discrete tokenization and dynamic multi-modal retrieval provides a new paradigm for streaming generative models, moving away from "look-ahead" dependencies and static conditioning toward adaptive, context-aware generation.