💻 computer science

Vorch-Director: Interactive World Story Model via Noise-Aware Error Rectification

Vorch-Director is an interactive world story model that enhances long-horizon audio-visual generation by introducing a noise-aware residual correction strategy to mitigate error accumulation and identity drift, while leveraging task embeddings and mixed-task training to support unified, multi-shot conditioning on the LTX-2 diffusion transformer.

Original authors: Lisai Zhang, Yidi Wu, Qi Liu, Xin Ma, Yang Ding, Gang Yue, Siqian Yang, Jingyuan Chen, Lin Ma, Yaohui Wang

Published 2026-08-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lisai Zhang, Yidi Wu, Qi Liu, Xin Ma, Yang Ding, Gang Yue, Siqian Yang, Jingyuan Chen, Lin Ma, Yaohui Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to tell a story. You give it a few sentences, and it writes the next one. Then, you take that new sentence, feed it back in, and ask for the next one. You keep doing this, hoping to build a whole novel. This is how many modern AI video generators work: they create a short clip, then use that clip as the "memory" to create the next one, stitching them together into a long movie. This process is called autoregressive continuation.

However, there is a tricky problem with this "copy-paste" method. When the robot learns, it is shown perfect, clean stories written by humans. But when it actually makes a movie, it has to rely on its own previous work, which is never quite perfect. It might have a tiny smudge of blur, a slightly wrong face, or a voice that sounds a bit off. When the robot uses that slightly flawed clip to make the next one, the errors get worse. By the time the movie is ten minutes long, the characters might look like melting wax, their voices might be garbled, and the story might drift into nonsense. This is known as exposure bias: the robot is trained on a pristine world but has to live in a messy one. Scientists have been trying to fix this so we can generate long, consistent, high-quality videos with synchronized sound, but until now, the solutions have been like trying to fix a leaky boat by just bailing out water without fixing the hole.

Enter Vorch-Director, a new approach that acts like a smart editor for AI video generation. The researchers behind this project realized that the way AI models make mistakes isn't random; it depends on how "noisy" or uncertain the model is at any given moment. Think of it like drawing a picture: when you are sketching the rough outline, your mistakes are big and structural (like getting the head too big). When you are adding the final details, your mistakes are tiny (like a crooked eyebrow). Previous methods tried to fix errors by randomly throwing "practice mistakes" at the AI during training, but they didn't care what kind of mistake it was. They might have tried to fix a tiny eyebrow error with a giant head-sized correction, which just made things worse.

Vorch-Director changes the game by being noise-aware. The system keeps a library of the AI's own mistakes, but it tags each mistake with the specific "noise level" it was made at. When the AI is learning to generate a new scene, the system looks at how uncertain the AI currently is. If the AI is in a "rough sketch" phase (high noise), the system feeds it big, structural errors to practice fixing. If the AI is in a "fine detail" phase (low noise), it feeds it tiny, subtle errors. This ensures the AI learns to correct the right kind of problems at the right time.

To make this work for long stories, the team also introduced a few clever tricks. They created a "clean anchor"—a short, perfect clip at the start of every new segment that the AI can look at to remember who the characters really are, preventing them from drifting into weird shapes. They also gave the AI special "name tags" (called task embeddings) so it knows exactly which part of the input is the story so far, which part is a reference photo, and which part is the new scene it needs to create. This helps the AI keep characters looking the same even when the camera cuts to a new angle or a new location.

The results are promising. When tested on benchmarks designed to measure how well AI keeps characters and stories consistent over time, Vorch-Director showed significantly less "drift" than other top models. In tests involving minute-long videos with synchronized audio, the new model kept faces recognizable and voices steady, whereas other models often resulted in characters morphing into strange blobs or voices becoming unintelligible. While the paper notes that there is still room for improvement—especially in matching the consistency of visual-only models—the new method suggests that by matching the type of error to the stage of generation, we can teach AI to tell much longer, more coherent, and more realistic stories without losing its mind along the way.

Technical Summary: Vorch-Director

Problem Statement

Autoregressive continuation offers a pathway to generate minute-scale, multi-shot audio-visual content by repeatedly extending a short-window generator. However, a critical train-test mismatch (exposure bias) hinders this approach. During training, models are conditioned on clean ground-truth histories (teacher forcing), whereas at inference, they rely on their own previously generated outputs. This recursive reuse of imperfect predictions leads to the accumulation of errors, manifesting as identity drift, over-smoothing, motion inconsistency, and audio-visual desynchronization over long temporal horizons.

Existing solutions attempt to mitigate this via memory retrieval (which assumes clean context) or self-rollout training (which is computationally expensive). Recent lightweight approaches, such as Matrix-Game, inject prediction residuals into the conditioning history to simulate inference-time errors during training. However, these methods overlook a fundamental property of diffusion models: prediction residuals are inherently dependent on the flow-matching noise level ( $\sigma$ ) at which they are produced. High- $\sigma$ residuals represent coarse structural deviations, while low- $\sigma$ residuals capture fine-detail errors. Indiscriminately reusing residuals across different noise regimes results in either under-corruption or over-corruption of the history, failing to faithfully replicate the inference distribution.

Additionally, long-horizon generation introduces positional extrapolation issues. Standard continuation models rely on continuous positional indices that grow with generation length, forcing the model to extrapolate beyond training ranges. This creates a second source of exposure bias.

Methodology: Vorch-Director

Vorch-Director addresses these challenges through a noise-aware error rectification strategy built upon the LTX-2 audio-visual Diffusion Transformer (DiT).

1. $\sigma$ -Aware Residual Injection

The core innovation is a strategy that aligns injected errors with the denoising process:

Residual Collection: During training, reconstruction residuals ( $\delta = \hat{x}_0 - x_0$ ) are collected from target tokens and stored in a ring buffer. Crucially, each residual is tagged with the noise level $\sigma$ at which it was generated.
Noise-Matched Sampling: When perturbing the conditioning history for a specific denoising step with noise level $\sigma$ , the system samples residuals from the buffer where the stored noise level $\sigma'$ satisfies $|\sigma' - \sigma| \leq \tau_\sigma$ .
Injection: These matched residuals are injected into the history tokens with a strength $\gamma$ (sampled from $U(0.9, 1.2)$ ). This ensures the "corrupted" history presented to the model during training mimics the specific error regime the model is currently learning to denoise, rather than applying arbitrary noise.

2. Conditioning Stabilization and Task Embeddings

To prevent total collapse from error injection and handle multi-shot complexity:

Clean Conditioning Sink: A short, uncorrupted window (1 second) of the history is appended as a "sink" ( $\tau_{sink}$ ). This provides a stable, clean anchor for the model to reference, preventing identity drift while the rest of the history ( $\tau_{hist}$ ) is corrupted.
Task Embeddings: To solve positional extrapolation, the model uses lightweight task embeddings ( $e_\tau$ ) to distinguish token roles (Target, History, Sink, Reference Images, Reference Audio) rather than relying solely on growing positional indices. This allows positional indices to be reset for new segments while preserving temporal consistency, decoupling semantic roles from absolute position.
Unified Prompt Template: A structured prompt template binds subject references (images and audio) and shot descriptions, enabling a single model to handle text-to-video, image-to-video, and audio-visual continuation tasks.

3. Training Objective

The model is trained on a four-way mixed-task corpus (25% each):

Video continuation (history only).
Continuation + Image (history + subject references).
Subject-IP (reference-image-driven generation).
Audio-visual continuation (history video/audio + references).

The loss function is a masked flow-matching loss applied only to the target tokens, ensuring the supervision signal remains clean while the context distribution matches inference conditions.

Key Contributions

$\sigma$ -Aware Residual Injection: A strategy that matches prediction residuals to the corresponding flow-matching noise level, substantially mitigating exposure bias in autoregressive diffusion generation.
Unified Conditioning Interface: A lightweight task-embedding system that distinguishes historical video, reference images, and target video, enabling positional encoding reuse across continuation steps and eliminating positional extrapolation bias.
Integrated Framework: An LTX-2-based framework combining the proposed training strategy with mixed-task training, supporting multi-shot, multi-subject, and reference-guided audio-visual generation within a single model.
New Benchmarking: Evaluation on ST-Bench and the introduction of a new long-horizon audio-visual benchmark featuring Anchor-relative Consistency (ARC) and Reappear metrics, which better characterize progressive degradation than standard adjacent-shot evaluations.

Results

Extensive experiments demonstrate that Vorch-Director improves stability and audio-visual fidelity over strong baselines:

ST-Bench (Multi-shot Story): Vorch-Director leads all four cross-shot consistency metrics by a wide margin. Notably, it achieves a ViCLIP score of 0.7887 compared to the best baseline (JoyAI-Echo) at 0.5900. It also significantly outperforms baselines on long-range identity measures (ARC: 0.6570 vs. 0.3961; Reappear: 0.6648 vs. 0.4114).
UnityShots (T2V & R2V): In Text-to-Video (T2V) settings, Vorch-Director leads in ViCLIP, ARC, and Reappear. In Reference-to-Video (R2V), it leads the strongest baseline on all four consistency metrics.
Long Audio-Visual Benchmark: In a 22-shot, minute-scale generation task with synchronized audio:
- Vorch-Director improves every cross-shot consistency metric against the audio-visual baseline JoyAI-Echo (e.g., ViCLIP 0.6269 vs. 0.4922).
- While visual-only baselines (StoryMem, Memento) show higher scores on some similarity metrics, the paper notes these systems generate no audio (solving a strictly easier problem) and exhibit less inter-frame motion, which inflates similarity scores.
- Qualitatively, Vorch-Director maintains protagonist recognizability and scene fidelity across 22 shots without the chromatic corruption or identity collapse seen in baselines like Memento and StoryMem.

Significance and Claims

The paper claims that the primary obstacle to minute-scale audio-visual generation is not context length itself, but the history mismatch caused by exposure bias. By aligning injected errors with the denoising process via noise-level awareness, Vorch-Director produces more realistic autoregressive histories while retaining the efficiency of teacher-forcing training.

The authors position Vorch-Director as an orthogonal improvement to existing memory and gating mechanisms; rather than deciding what history to recall, it makes the base continuation model robust to imperfect recalled or generated history. The work demonstrates that a single model can support complex, multi-shot, multi-subject, and synchronized audio-visual generation, addressing the drift and desynchronization issues that currently limit autoregressive video generation. The paper modestly notes that closing the gap to visual-only systems while maintaining synchronized audio remains an open challenge, but Vorch-Director represents a significant step toward stable, long-horizon generation.