Prompt Relay: Inference-Time Temporal Control for Multi-Event Video Generation

1. Problem Statement

Current video diffusion models excel at generating high-quality, single-event videos but struggle with multi-event temporal control. When users provide a single paragraph describing a sequence of events (e.g., "A man eats pasta, then a woman walks by"), existing models suffer from semantic entanglement. Because cross-attention mechanisms treat the entire prompt as global context for every frame, semantic concepts intended for specific time segments "bleed" into one another. This results in:

• Poor Temporal Alignment: Events occur at the wrong times or simultaneously.
• Lack of Explicit Control: Users cannot specify the exact start time, duration, or order of events.
• Visual Artifacts: Attempts to force temporal control via hard masking often create discontinuities at segment boundaries, causing flickering or unnatural morphing.

Existing solutions either require extensive retraining with annotated data (shifting the model's distribution) or impose rigid structural constraints that limit generality.

2. Methodology: Prompt Relay

The authors propose Prompt Relay, a training-free, plug-and-play, inference-time method that enables fine-grained temporal control without modifying the model architecture or adding computational overhead.

Core Mechanism: Temporal Cross-Attention Routing

The method operates by modifying the cross-attention logits within the diffusion model. Instead of allowing every video token (query) to attend equally to all text tokens (keys) throughout the video, Prompt Relay routes each textual prompt to its designated temporal interval.

1. Input Structure: The user provides a sequence of temporally constrained prompts: $\{(p_s, t_{start}^s, t_{end}^s)\}_{s=1}^N$, where $p_s$ is the text prompt and $[t_{start}^s, t_{end}^s]$ is the time segment.
2. Attention Penalty: A penalty term $C(Q, K)$ is subtracted from the attention logits before the Softmax operation:
   $$\text{Attn} = \text{Softmax}\left(\frac{QK^T}{\sqrt{d}} - C(Q, K)\right)V$$
   This penalty suppresses attention between a query token (video frame) and a key token (text prompt) if they do not belong to the same temporal segment.

Boundary-Attention Decay

A key innovation is the Boundary-Attention Decay mechanism, which replaces "hard masking" with a smooth Gaussian penalty.

• The Problem with Hard Masking: Simply blocking attention outside a segment ($C = -\infty$) creates a sharp discontinuity. The cross-attention switches abruptly while self-attention remains continuous, forcing the model to reconcile conflicting signals and causing visual glitches.
• The Solution: Prompt Relay applies a soft Gaussian penalty that decays smoothly near segment boundaries.
  • Free-Attention Window: Inside a central window around the segment midpoint ($w$), no penalty is applied.
  • Smooth Decay: Outside this window, attention is attenuated based on the temporal distance from the segment midpoint.
  • Co-activation: Near boundaries, the model is allowed to "co-activate" neighboring prompts, creating a gradual handoff region where the transition is planned jointly before being committed to the visual representation.

The penalty function is defined as:
$$C(i, j) = \frac{\text{ReLU}(|f(i) - m_s| - w)^2}{2\sigma^2}$$
Where $f(i)$ is the frame index, $m_s$ is the segment midpoint, and $\sigma$ is tuned to ensure the attention prior drops to a negligible value $\epsilon$ (e.g., 0.1) at the segment endpoints.

3. Key Contributions

1. Prompt Relay Framework: A novel, inference-time routing mechanism that enables multi-event video generation with fine-grained temporal control, requiring zero training and zero architectural changes.
2. Boundary-Attention Decay: A soft Gaussian penalty strategy that prevents semantic interference while avoiding the visual artifacts caused by hard masking, ensuring smooth transitions between events.
3. Global Coherence: The method supports a "global prompt" to maintain persistent context (e.g., character identity, scene style) across segments, mitigating the drift often seen in segmented generation.

4. Experimental Results

The method was evaluated on the state-of-the-art Wan2.2-T2V-A14B model and compared against baselines including Sora Storyboard, Veo 3.1, Kling 2.6, and the base Wan 2.2.

• Quantitative Evaluation (Human Preference):

  • Temporal Prompt Alignment: Prompt Relay achieved the best score (1.10, where lower is better), significantly outperforming baselines like Wan 2.2 (4.00) and Veo 3.1 (3.93).
  • Transition Naturalness: It achieved the highest rating (1.17), demonstrating superior smoothness at segment boundaries compared to hard masking approaches.
  • Visual Quality: While Kling 2.6 and Veo 3.1 retained slightly higher overall visual fidelity (due to their stronger backbones), Prompt Relay improved the visual quality of the Wan 2.2 baseline (2.83 vs. 4.00) by reducing semantic competition in the attention space.

• Qualitative Observations:

  • Prompt Relay successfully generated complex sequences (e.g., a camera zooming into an eagle's eye to reveal a cyberpunk city, then zooming out to a living room) without semantic bleeding.
  • It handled rapid camera movements and scene changes (e.g., caveman to Spartan to Knight) with coherent transitions.

5. Significance and Limitations

Significance:

• Movie-Grade Synthesis: Prompt Relay addresses a critical gap in video generation, moving from single-scene synthesis to coherent storytelling with precise timing and ordering.
• Efficiency: As a training-free, plug-and-play module, it can be immediately applied to existing pre-trained models without the cost of retraining or data annotation.
• Mechanism Insight: It demonstrates that temporal control can be achieved by manipulating attention routing rather than retraining the backbone, offering a new direction for controllable generative AI.

Limitations:

• Persistent Elements: Without a global prompt, persistent visual elements (characters, objects) may drift across segments because they are not explicitly shared. The authors note this is solvable by including a global context prompt.
• Backbone Dependency: The ultimate visual fidelity is still bounded by the capacity of the underlying video diffusion model (e.g., Wan 2.2 vs. Veo 3.1).

In conclusion, Prompt Relay represents a pivotal step toward controllable, narrative-driven video synthesis by solving the problem of semantic entanglement through a simple yet effective attention-level routing mechanism.