Making Video Models Adhere to User Intent with Minor Adjustments

1. Problem Statement

Text-to-video (T2V) diffusion models have achieved significant advancements in generation quality, but controlling spatial object placement and trajectories remains a major challenge.

• The Gap: While methods exist to inject control signals (e.g., bounding boxes) into pre-trained models without retraining (e.g., Trailblazer), these approaches often suffer from a mismatch between the control signal and the model's internal representations.
• The Consequence: Enforcing strict adherence to user-provided bounding boxes often leads to:
  • Degraded video quality: Artificial artifacts, over-saturation, or loss of background details.
  • Poor adherence: The model fails to follow the trajectory because the injected control disrupts the natural attention flow of the diffusion model.
• Core Hypothesis: The paper posits that user-provided bounding boxes are often slightly misaligned with the model's internal "familiarity" (attention maps). By making minor, optimized adjustments to these boxes, one can significantly improve both generation quality and control adherence without retraining the model.

2. Methodology

The authors propose a training-free optimization framework that adjusts user-provided bounding boxes to align with the internal cross-attention mechanisms of video diffusion models. The method consists of three key components:

A. Differentiable Attention Map Editing

Existing methods (like Trailblazer) use binary masks to replace or modify attention maps. This creates discontinuous boundaries, making the process non-differentiable with respect to the box coordinates, which prevents gradient-based optimization.

• Solution: The authors introduce smooth masks using 1D sigmoid step functions to create soft transitions at the box boundaries.
• Mechanism: Instead of a hard binary mask ($M \in \{0, 1\}$), they use a continuous mask ($M_B$) derived from a Gaussian map smoothed by sigmoid functions. This allows the attention map editing operation to be fully differentiable with respect to the bounding box parameters ($b_{left}, b_{top}, b_{right}, b_{bottom}$).

B. Attention Maximization Objective

The goal is to find a bounding box position that the neural network "prefers" (i.e., where its attention naturally focuses).

• Forward Propagation: The method edits the attention map at layer $l$ and observes the resulting attention map at the next layer ($l+1$).
• Loss Function ($L_{attn}$): The optimization maximizes the sum of attention values within the bounding box at layer $l+1$. This ensures the edit is "easy" for the network to follow.
  $$L_{attn} = \left\| 1 - \frac{\sum (A^{l+1}_* \odot M^{l+1}_*)}{\sum A^{l+1}_*} \right\|_2^2$$

C. Balancing and Regularization

To prevent the model from ignoring the background or deviating too far from the user's intent, two additional loss terms are introduced:

1. Balancing Loss ($L_{\neg attn}$): Ensures the model still attends to the region outside the box, preserving background context and prompt fidelity.
2. Regularization Loss ($L_{reg}$): Penalizes large deviations from the original user-provided box ($b_{user}$), ensuring the adjustment remains a "minor" modification.
   $$L_{total} = L_{attn} + \lambda_{\neg attn}L_{\neg attn} + \lambda_{reg}L_{reg}$$

The bounding box parameters are optimized using the Adam optimizer over a few iterations (e.g., 25 total steps) before generating the final video.

3. Key Contributions

1. Demonstration of Sensitivity: The paper establishes that small adjustments to user intent (bounding boxes) can lead to significant improvements in both video quality and control adherence.
2. Differentiable Pipeline: A novel method to make non-differentiable attention editing (common in training-free control) fully differentiable with respect to spatial control parameters.
3. Balanced Optimization: An objective function that maximizes attention within the box while simultaneously preserving background attention and staying close to the original user input.
4. Training-Free Approach: The method works with off-the-shelf video diffusion models (e.g., Trailblazer, T2V-Turbo) without requiring additional model training or fine-tuning.

4. Experimental Results

The method was evaluated on the Animal Kingdom dataset using two backbone models: Trailblazer and T2V-Turbo.

• Quantitative Metrics:
  • Human Preference: The method achieved higher PickScore and HPS v2 (Human Preference Score) compared to baselines like Peekaboo, Freetraj, and the original Trailblazer.
  • Control Accuracy: The method maintained competitive mIOU (Mean Intersection over Union) scores, proving that the adjustments did not degrade the strictness of the control.
• Qualitative Results:
  • Generated videos showed fewer artifacts, better motion coherence, and more faithful adherence to the prompt (e.g., "The jellyfish is swimming") compared to baselines.
  • Baselines often produced "cut-off" objects or ignored the background; the proposed method balanced foreground and background attention effectively.
• User Study:
  • A study with 20 participants showed the proposed method was significantly preferred over baselines for both generation quality and trajectory faithfulness.
  • Even when applying the optimized boxes to the baseline Trailblazer method, performance improved, validating that the box adjustment itself is the key factor.

5. Significance and Impact

• Paradigm Shift: The paper challenges the assumption that user inputs must be rigidly enforced. Instead, it suggests that optimizing control signals to match the model's internal priors yields superior results.
• Efficiency: As a training-free method, it offers a computationally efficient way to improve control in existing large-scale video models.
• Generalizability: While focused on bounding boxes, the authors argue this principle (aligning control signals with internal representations) applies to other modalities like trajectories, sketches, or depth cues.
• Limitations: The method requires partial back-propagation during the optimization phase, increasing generation time (approx. 1m 37s vs. 48s for baseline). Additionally, if the underlying video model lacks knowledge of the specific object/prompt, the method cannot force generation (failure cases shown).

In conclusion, this work provides a foundational insight that control signals in generative AI are sensitive to misalignment, and minor, differentiable optimizations can bridge the gap between user intent and model capability.