ASTRA: Let Arbitrary Subjects Transform in Video Editing

1. Problem Statement

While recent generative models have advanced video editing, existing methods face critical limitations in complex, multi-subject scenarios (dense scenes with heavy occlusions).

• Attribute Leakage & Boundary Entanglement: Current approaches struggle to maintain distinct boundaries between interacting subjects. This leads to "bleeding" where attributes of one subject (e.g., a dog's head) incorrectly transfer to another (e.g., a robot wolf's body).
• Attention Dilution: Naive text prompts fail to provide sufficient constraints for multiple subjects simultaneously, causing the model to lose focus on specific targets or fail to execute the intended transformation for all subjects.
• Temporal Instability: In dense scenes, mask boundaries often drift or flicker over time, causing structural artifacts and identity drift.
• Dependency on Fine-tuning: Many state-of-the-art (SOTA) methods require task-specific fine-tuning or rigid guiding masks, limiting their generalization to open-domain videos.

2. Methodology: The ASTRA Framework

ASTRA is a training-free, plug-and-play framework designed to edit an arbitrary number of subjects while preserving background integrity. It integrates two core modules into a pretrained mask-driven video generation backbone (specifically Wan2.1).

A. Prompt-Guided Multimodal Alignment Module

To solve the issue of insufficient semantic constraints and attention dilution:

1. Visual Prior Extraction: The system extracts subject-specific tokens from the user's editing prompt and queries a pretrained Text-to-Image (T2I) model (SDXL) to generate a reference image ($I_{ref}$) representing the target appearance.
2. Multimodal Synthesis: A Vision-Language Model (VLM) takes the original editing prompt ($P_{edit}$) and the reference image ($I_{ref}$) to generate an expanded, grounded textual prompt ($P_{target}$). This process explicitly aligns abstract text with concrete visual priors.
3. Condition Injection: The enriched text ($P_{target}$) is injected via cross-attention, and the reference image is prepended as the first frame of the input sequence. This ensures the model has a strong visual anchor for every subject, preventing semantic drift.

B. Prior-Based Mask Retargeting Module

To address mask boundary entanglement and temporal inconsistency:

1. Depth-Guided Re-estimation: Instead of relying solely on initial segmentation masks, the module utilizes depth priors (from Depth Anything V2) to re-estimate instance boundaries.
2. Soft Mask Generation: The initial binary masks are processed through morphological dilation and Gaussian filtering to create "soft" masks. This smooths transitions and reduces hard-edge artifacts.
3. Feature Alignment: The module aligns latent features from the mask and depth conditions. It uses a weighted combination where depth features guide the geometry of the target subjects (handling occlusions), while mask features constrain the background.
4. Temporal Consistency: This process generates a temporally coherent sequence of retargeted masks that strictly adhere to the source video's dynamics.

C. Timestep-Aware Video Generation

ASTRA employs a phased injection strategy to balance structural control and detail synthesis:

• Early Steps ($t \le \tau$): The retargeted features (combining multimodal alignment and mask retargeting) are injected to establish the global structure, layout, and occlusion order.
• Late Steps ($t > \tau$): The system reverts to standard mask-only conditioning to refine high-frequency details and textures, preventing the structural constraints from corrupting fine visual quality.
• Threshold ($\tau$): Empirically set to 30 (out of 50 steps) to optimize the trade-off between structural stability and detail.

3. Key Contributions

1. ASTRA Framework: A novel, training-free solution that enables seamless transformation of an arbitrary number of subjects without model retraining.
2. Dual-Module Architecture:
   • Multimodal Alignment: Solves attention dilution by grounding text prompts with visual priors via VLMs.
   • Mask Retargeting: Solves boundary entanglement by using depth priors to generate temporally consistent, soft masks.
3. Plug-and-Play Compatibility: The framework acts as a versatile conditioning layer compatible with various mask-driven video generators (e.g., Wan2.1, CogVideo), significantly boosting their performance.
4. MSVBench: The authors introduced a new benchmark comprising 100 challenging multi-subject video sequences with dense layouts and heavy occlusions to facilitate standardized evaluation in this domain.

4. Experimental Results

Extensive experiments were conducted on MSVBench and the LOVEU-TGVE-2023 benchmark.

• Quantitative Performance: ASTRA achieved State-of-the-Art (SOTA) results across all metrics:
  • Q-Edit (Editing Fidelity): 14.72 (vs. 13.24 for the next best open-source method, VACE).
  • CLIP-T (Text Alignment): 27.23.
  • CM-Err (Layout/Count Consistency): 2.83 (lower is better), indicating superior preservation of subject counts and spatial layouts.
  • Warp-Err (Background Preservation): 1.85.
• Comparison with Commercial Models: ASTRA outperformed closed-source commercial systems like Runway and Kling in editing fidelity and layout consistency.
• Efficiency: Unlike iterative editing methods that require multiple passes (taking ~18 mins), ASTRA operates in a single forward pass, completing editing in approximately 10 minutes.
• Generalization: The framework demonstrated robust performance across different backbone architectures (Wan2.1, CogVideo) and diverse editing scenarios (face swapping, clothing changes, long videos).

5. Significance

ASTRA represents a significant leap in controllable video generation. By decoupling the editing logic from the generative backbone, it provides a universal solution for complex multi-subject editing—a task previously plagued by attribute leakage and instability.

• Practical Impact: It lowers the barrier for high-quality video editing, removing the need for laborious manual modeling or expensive fine-tuning.
• Scientific Contribution: It establishes a new paradigm for handling dense scenes by explicitly reconciling competing semantic signals through multimodal alignment and depth-aware mask retargeting.
• Future Directions: The authors note that while effective, the reliance on external depth/mask priors can introduce artifacts in extreme occlusions, and the computational cost of DiT backbones limits real-time application, suggesting future work in feature distillation.