Anchored Video Generation: Decoupling Scene Construction and Temporal Synthesis in Text-to-Video Diffusion Models

1. Problem Statement

Current State-of-the-Art (SoTA) Text-to-Video (T2V) diffusion models, despite achieving high visual fidelity, suffer from significant compositional failures and logical temporal inconsistencies. The authors argue that these errors often stem from the model's inability to construct a semantically correct and logically consistent initial frame.

• The Core Issue: Existing T2V models attempt to jointly learn static scene construction (what the scene looks like) and temporal synthesis (how the scene moves) within a single, over-entangled architecture.
• Failure Modes: When a prompt requires complex logic (e.g., "a chicken tasting sauce"), the model often misinterprets the initial state (e.g., generating a "roasted chicken" instead of a live one). Once the initial scene is semantically incorrect, the temporal motion model cannot recover, leading to prompt violations regardless of the motion strength.
• Diagnostic Insight: Analysis reveals that T2V models often produce distributionally shifted frames (poor scene fidelity) yet achieve reasonable motion scores (FVD). Conversely, Image-to-Video (I2V) models have high scene fidelity but weaker temporal coherence. This suggests a structural mismatch where current T2V models fail to ground the scene before animating it.

2. Methodology: Anchored Video Generation (AVG)

The authors propose Anchored Video Generation (AVG), a modular pipeline that decouples video generation into three specialized stages to address scene construction and temporal synthesis independently.

Stage 1: Reasoning (Prompt Rewriting)

• Goal: Resolve temporal ambiguities in the user prompt.
• Process: A Large Language Model (LLM) rewrites the original video prompt into a specific description of the initial scene state only. It removes temporal transitions (e.g., "blooming," "rusting") and focuses purely on spatial layout, object appearance, and configuration.
• Output: A refined prompt ($y_0$) describing the static starting frame.

Stage 2: Composition (Anchor Generation)

• Goal: Synthesize a high-quality, semantically grounded anchor frame.
• Process: A Text-to-Image (T2I) model uses the refined prompt ($y_0$) to generate a high-quality image ($I_a$). This image serves as the "visual anchor" or the ground-truth starting state for the video.
• Flexibility: Users can optionally intervene to select preferred visual styles for the anchor.

Stage 3: Temporal Synthesis (Anchored Generation)

• Goal: Animate the scene while strictly adhering to the anchor.
• Mechanism: A Video Diffusion Model (VDM) is finetuned to accept a visual anchor as a hard constraint.
  • Anchor-Grounding Finetuning: The VDM is trained using LoRA (Low-Rank Adaptation). During training, a clean latent of the anchor image is injected into the noisy video latent sequence at a specific frame (usually $t=0$) with a diffusion timestep of 0, while other frames remain noisy at timestep $t$. This teaches the model to treat the anchor as a fixed visual constraint.
  • Inference: During generation, the anchor image is encoded and injected into the first frame of the latent sequence with $t=0$. The model then denoises the remaining frames, focusing its capacity entirely on generating coherent temporal dynamics consistent with the anchor.

3. Key Contributions

1. Diagnostic Insight: Identified that the primary failure mode in T2V is the inability to construct a correct initial scene, leading to a decoupling of scene construction and temporal modeling as a necessary design principle.
2. AVG Pipeline: Introduced a simple, modular three-stage pipeline (Reasoning, Composition, Temporal Synthesis) that significantly improves scene grounding without requiring massive architectural changes.
3. Efficient Finetuning: Demonstrated that lightweight LoRA finetuning is sufficient to teach existing T2V backbones (like Wan 2.2, CogVideo) to accept visual anchors, avoiding the need for full retraining.
4. Efficiency Gains: Showed that visual anchoring stabilizes the diffusion trajectory, allowing for a 70% reduction in sampling steps (e.g., from 50 to 15) without performance degradation, resulting in a 2.1x speedup in wall-clock time.

4. Experimental Results

The authors evaluated AVG on T2V-CompBench (compositional control) and VBench 2.0 (holistic quality) across multiple models (Wan 2.2 5B/14B, CogVideo 1.5 5B, Wan 2.1 1B).

• Compositional Performance (CompBench):

  • AVG achieved a new State-of-the-Art (SOTA) on CompBench.
  • Wan 5B (Anchored) outperformed the larger Wan 14B (Standard T2V) and even the proprietary PixVerse-V3.
  • The anchored Wan 5B achieved a +41.98% relative improvement over its baseline T2V version.
  • The anchored Wan 14B reached performance comparable to its own pretrained I2V variant (0.661 vs 0.666) but with much less training cost.

• Holistic Quality (VBench 2.0):

  • Significant improvements in Composition, Controllability, and Physics across all tested models.
  • Wan 5B (Anchored) improved the average score by +15.3%.
  • Note: There was a slight decline in "Human Fidelity" (specifically identity preservation), which the authors attribute to the difficulty of maintaining identity across actions, a separate challenge from scene composition.

• Robustness & Efficiency:

  • Sampling Steps: Reducing steps from 50 to 15 caused a massive performance drop (-17%) for standard T2V, but the Anchored model remained stable (+0.3% variation).
  • Diversity: The pipeline maintains diversity comparable to baselines when varying seeds, though fixing the anchor reduces motion-level diversity (as expected).

5. Significance and Implications

• Paradigm Shift: The paper challenges the "end-to-end" scaling approach in T2V. It suggests that better grounding (via visual anchors) is more effective than simply increasing model size or training data for compositional tasks.
• Practicality: The method is a "lightweight retrofit" for existing models, requiring only LoRA finetuning and an LLM/T2I front-end, making it highly accessible.
• Evaluation Gap: The results highlight that current benchmarks may conflate scene construction errors with temporal modeling failures. Future evaluations should consider anchor-conditioned baselines to isolate true temporal reasoning capabilities.
• Future Directions: The authors suggest that while AVG solves scene composition, future work must address identity preservation and define new diversity metrics that distinguish between "correct diversity" (valid variations) and "incorrect diversity" (hallucinations).

In conclusion, Anchored Video Generation provides a robust, efficient, and controllable framework for text-to-video synthesis by explicitly separating the "what" (scene construction) from the "how" (temporal evolution), effectively solving a critical bottleneck in current generative video models.