Technical Summary: Compositional Video Generation via Inference-Time Guidance (CVG)

Problem Statement

Text-to-video (T2V) diffusion models have achieved significant progress in visual fidelity and temporal coherence. However, they frequently fail on prompts requiring fine-grained compositional understanding, such as specific spatial relations (e.g., "behind," "between"), motion directions (e.g., "turning left"), numerical counts, and complex interactions between multiple entities. While standard text conditioning and classifier-free guidance provide global prompt signals, they often lack the explicit verification needed to satisfy these fine-grained constraints. Existing solutions to improve compositional control typically require retraining the generator with auxiliary inputs (e.g., bounding boxes, layouts), fine-tuning via reward models, or expensive test-time search, all of which demand additional data, compute, or user-provided controls.

Methodology: CVG

The authors propose CVG (Compositional Video Generation), an inference-time guidance framework that improves compositional faithfulness in frozen T2V models without modifying their architecture or weights. The core hypothesis is that the internal cross-attention maps of a pretrained model already encode the necessary spatiotemporal grounding signals to evaluate compositional correctness.

1. Core Mechanism

CVG operates by extracting subject-token cross-attention maps from the frozen T2V generator during the denoising process. These maps indicate where the model localizes specific entities (e.g., "dog," "ball") and how these localizations evolve over time. Instead of using the relation token itself, the classifier receives only the attention maps of the entities involved in the composition.

2. The Compositional Classifier

A lightweight classifier is trained to predict compositional labels (e.g., "left," "behind") based on these attention maps.

• Architecture: The classifier consists of a learned aggregation module, a frozen Vision-Language Model (VLM) backbone (Qwen2.5-VL-7B), and a trainable classification head.
• Training Data: The classifier is trained on a mix of real videos (obtained via video inversion) and synthetic videos.
• Dual Inversion for Leakage Prevention: To prevent "composition leakage" (where the classifier cheats by reading textual traces of the relation word in the attention maps rather than visual grounding), the authors employ a dual inversion strategy. Each training video is inverted twice: once with the correct composition prompt and once with an incorrect prompt sampled from the vocabulary. Both inversions are supervised with the same ground-truth label. This forces the classifier to rely on the actual visual spatiotemporal arrangement of the entities.
• Generalization: Because the classifier is built on a frozen VLM backbone, it inherits broad semantic priors, allowing it to generalize to semantically related composition labels beyond the specific set seen during training.

3. Inference-Time Guidance

During video generation, the classifier acts as a differentiable controller:

1. At selected denoising steps, the system extracts subject-token cross-attention maps from the current latent.
2. The classifier predicts a distribution over compositional classes.
3. A cross-entropy loss is computed between the prediction and the target composition label.
4. The gradient of this loss is backpropagated to update the latent video variable, steering the trajectory toward the desired composition.
5. Guidance Schedule: To preserve the visual quality and appearance priors of the underlying generator, guidance is applied only during the first 8 denoising steps, where coarse motion and scene structure are established.

4. Extensions

• Multi-Composition Guidance: CVG supports temporally structured prompts (e.g., "turn left then right") by applying specific composition constraints to masked temporal segments of the video.
• Autoregressive Generation: The method extends to long-video generation by assigning target composition labels to individual generated batches while maintaining temporal continuity through the autoregressive context.

Key Contributions

1. Inference-Time Control Framework: CVG provides a method to steer frozen T2V generators using gradients from a lightweight compositional classifier, eliminating the need for fine-tuning, auxiliary conditioning, or user-supplied layouts.
2. VLM-Based Classifier: The introduction of a classifier trained on subject-token cross-attention maps using a frozen VLM backbone, which enables generalization to unseen or rare compositional configurations.
3. Dual Inversion Training: A novel training strategy using dual inversion to mitigate composition leakage, ensuring the classifier learns from visual grounding rather than textual artifacts.
4. Empirical Validation: Comprehensive evaluation demonstrating improved compositional faithfulness while preserving visual quality across multiple backbones (Wan2.2, CogVideoX) and generation regimes (short and long videos).

Experimental Results

The authors evaluated CVG on the T2V-CompBench benchmark and VBench (for short videos) and VBench-Long (for long videos), comparing against base generators and TTOM (a strong layout-guided test-time optimization baseline).

• Quantitative Performance: CVG consistently outperformed both base models and TTOM across all tested backbones.
  • On Wan2.2-14B, CVG improved the T2V-CompBench average score from 0.556 (base) to 0.594, surpassing TTOM (0.579).
  • On VBench, CVG achieved the highest Final Scores for short videos (e.g., 86.3% on Wan2.2-14B vs. 85.0% for TTOM) and long videos (69.62% vs. 68.08% for TTOM).
  • Gains were most pronounced in composition-sensitive dimensions: Motion, Spatial Relations, Content Attributes, and Dynamic Attributes.
• Qualitative Improvements: Visual examples showed CVG correctly realizing complex constraints (e.g., "rabbit on top of turtle," "crab crawling from beneath driftwood") where baselines and TTOM failed with incorrect spatial relations or motion directions.
• User Study: In a human preference study on 100 prompts, participants preferred CVG over TTOM for Compositionality (39.6% vs. 20.7% on Wan2.2-14B) and Text Alignment, while rating Quality as comparable in the majority of cases (54.6% "both").

Significance and Claims

The paper claims that compositional failures in T2V generation need not be addressed by retraining the generator. Instead, the model's own internal grounding signals (cross-attention maps) can be leveraged via a lightweight, frozen-VLM-based classifier to steer generation at inference time.

The authors emphasize that CVG achieves these improvements without:

• Modifying the model architecture.
• Fine-tuning the generator.
• Requiring user-supplied layouts, bounding boxes, or spatial controls.

The work suggests that internal representations of video diffusion models can serve as actionable control mechanisms for aligning generated videos with complex compositional prompts, offering a training-free and efficient path to higher compositional faithfulness.

Limitations

The authors note that CVG's effectiveness depends on the quality of the model's internal grounding signals. While the VLM backbone aids generalization, the method is strongest for subject-centric compositions (motion direction, spatial layout) and may be less reliable for ambiguous prompts, subtle compositions, or scenes with multiple highly interacting subjects. The approach steers the denoising trajectory rather than learning new generative capabilities, meaning it cannot fix fundamental generative deficits in the base model.