Technical Summary: JoyAI-Echo-1.5

Problem Statement

The current frontier in video generation faces two primary challenges: achieving continuity and control in long-form narratives, and enabling precise interaction in interactive world modeling. Existing models are largely designed around fixed temporal windows, leading to identity drift, unstable self-generated histories, and an inability to translate heterogeneous action inputs into consistent motion. Specifically, long-form generation struggles to preserve character identities and speaker voices across shots, while interactive world models often fail to maintain geometric consistency or respond accurately to user navigation intents over extended rollouts.

Methodology

JoyAI-Echo-1.5 is a unified audio-visual generation system comprising two purpose-built variants: a Long-Video System for persistent stories and a World Model System for interactive environments. Both variants share a causal few-step generator architecture trained via a progressive curriculum.

1. Data Construction

The system employs a two-stage data curriculum:

• Stage I (Memory Construction): Constructs identity-centric corpora from recurring characters in scene-disjoint clips. This stage focuses on training audio-visual memory conditioning, pairing target shots with variable subsets of historical shots to learn robust identity and voice representations.
• Stage II (High-Quality Enhancement): Uses high-resolution, subtitle-free, and Chinese-enriched videos (single and multi-shot) to improve general audio-visual quality, spatial coverage, and transition modeling.
• Action Annotation: For the world model, heterogeneous navigation inputs (UE renders, internal gameplay, human gameplay) are unified into calibrated metric 6-DoF camera trajectories. A metric-depth-assisted pipeline recovers long-sequence poses to ensure scale consistency and robustness.

2. Long-Video Generation: Memory and Consistency

• Unified Audio-Visual Memory Conditioning: The system introduces composable cross-shot memory. Visual memory is constructed by sampling frames from multiple prior shots, while audio memory utilizes speech-filtered full-shot audio to derive robust speaker cues.
• RoPE Separation: To handle memory without timeline confusion, the model assigns separated temporal Rotary Positional Embedding (RoPE) regions to the current target, historical memory, and initial memory. This allows the model to jointly access appearance and voice evidence while keeping memory tokens distinct from the prediction timeline.
• Conditioning: The system supports flexible combinations of Text-to-Audio-Video (T2AV), Image-to-Audio-Video (I2AV), and memory-conditioned variants (MT2AV, MTI2AV) under a unified interface.

3. World Modeling: Geometric Control

• Controller-Agnostic Interface: The system converts diverse navigation inputs into a unified, subject-agnostic 6-DoF relative trajectory representation.
• Scale Calibration: A global scale factor is applied to calibrate relative translations across different data sources, ensuring consistent velocity semantics.
• Unified Camera Positional Encoding (UCPE): Trajectory geometry is injected via a parallel attention pathway using UCPE, making pairwise camera geometry invariant to global coordinate shifts while preserving calibrated relative translation magnitudes.

4. Training Pipeline: Causal Few-Step Generation

To enable efficient, stable long-horizon generation, the bidirectional audio-visual backbone is transformed into a causal few-step generator through a three-stage process:

• Audio-Visual Teacher Forcing: Initializes a causal streaming generator by training on ground-truth historical contexts, converting bidirectional attention into chunk-wise causal attention.
• Self-Gradient Forcing (SGF):
  • Short-Horizon SGF: Trains the model on self-generated rollouts by reconstructing the causal computation in a differentiable forward pass. This allows gradients to flow through the context-writing process, improving robustness to autoregressive errors.
  • Long-Horizon SGF: Extends SGF to longer trajectories using a "sink-plus-FIFO" policy to bound the KV cache while maintaining multi-hop gradient paths through layer-wise propagation.
• Distribution Matching Distillation (DMD): Converts the multi-step generator into an 8-step (long-video) or 4-step (world model) student.
  • Asymmetric Optimization: The video branch is fully fine-tuned to adapt to structural changes, while the audio branch uses Low-Rank Adaptation (LoRA) to preserve acoustic priors.
  • Coarse-to-Fine Scheduling: High-noise updates stabilize video structure, followed by low-noise updates to refine audio details.
  • Audio Constraints: Adversarial losses and RMS-level matching are introduced to prevent over-smoothing and energy drift in audio.

5. Agentic Planning and Control

A Director Agent orchestrates the generation process. It resolves user intent into a screenplay, manages memory retrieval, performs prompt enhancement, and handles shot-wise review and revision. This agent ensures that generation is conditioned on grounded visual evidence and user decisions, rather than unverified plans.

6. Arbitrary-Step Super-Resolution (SR)

A dedicated SR stage restores fine textures at delivery resolution. Using MeanFlow, the system distills a flow-matching teacher into a model that predicts average velocity over an interval. This allows for arbitrary-step sampling (e.g., 1-step to N-step) at inference time without retraining, serving both latency-critical and quality-focused use cases.

Key Contributions

1. A Long-Video System that Remembers: Introduces multi-shot audio-visual memory with speech-filtered voice representations and composable conditioning to preserve identity across shots.
2. A World Model that Responds: Establishes a controller-agnostic action interface based on calibrated metric 6-DoF trajectories, enabling precise camera control across heterogeneous data sources and viewpoints.
3. A Generator Built to Continue: Develops a causal few-step training pipeline combining audio-visual teacher forcing with short- and long-horizon Self-Gradient Forcing, allowing the model to learn from its own rollout distribution and remain stable over extended generation.

Results

• Long-Video Performance: JoyAI-Echo-1.5 achieves state-of-the-art performance on a 100-story benchmark, outperforming baselines in cross-shot visual consistency (ViCLIP: 0.8264), character identity (Self-CIDS: 0.7937), and speaker consistency (Voice: 0.8524). It also leads in text alignment and speech fidelity. Human evaluations show significant preference over HappyOyster in instruction following and audio-visual synchronization.
• World Model Performance: On the WBench Navigation split, the undistilled variant achieves the highest average score (81.7), while the distilled 4-step causal variant ranks second (81.0) with superior interaction scores (87.9). On SANA-WM-Bench, the system leads in trajectory following and visual quality for both short (241 frames) and long (961 frames) horizons. The distilled causal model demonstrates robust 60-second autoregressive rollouts with minimal drift in scene identity and structure.
• Qualitative Improvements: The system effectively preserves facial structure, clothing, and voice identity across temporally distant shots and maintains stable exposure and color balance in long-form sequences.

Significance

The paper posits that memory, geometric control, and rollout-aware training provide a practical foundation for generating coherent stories and continuously evolving interactive worlds. JoyAI-Echo-1.5 demonstrates that video generation can move beyond creating isolated clips toward sustaining narratives and responding to user intent with high fidelity. The system's ability to maintain consistency over long horizons and across heterogeneous control inputs addresses critical limitations in current video generation models, paving the way for more immersive and interactive media experiences. The authors note that while significant progress is made, accumulated rotational drift on difficult long-horizon trajectories remains a key limitation for future work.