Reshoot-Anything: A Self-Supervised Model for In-the-Wild Video Reshooting

1. Problem Statement

The core challenge in dynamic video reshooting (modifying camera trajectories in existing videos) is the severe scarcity of large-scale, paired multi-view video data for non-rigid scenes.

• Limitations of Existing Approaches:
  • Synthetic Data: Models trained purely on synthetic data suffer from a "sim-to-real" gap, failing to generalize to photorealistic videos or complex interactions (e.g., hitting a tennis ball), often producing visual artifacts.
  • Anchor-Only Methods: Approaches relying solely on 3D/4D reconstruction anchors (e.g., point clouds) are highly vulnerable to reconstruction errors. Artifacts in the anchor propagate directly to the output, causing texture drift and structural hallucinations.
  • Data Scarcity: Acquiring real-world videos capturing the exact same action from multiple synchronized camera angles is logistically intractable.

2. Methodology

The authors propose a self-supervised framework that generates training data from single monocular videos, bypassing the need for paired multi-view datasets.

A. Pseudo Multi-View Triplet Generation

Instead of using real multi-view data, the model synthesizes training triplets $(V_s, V_a, V_t)$ from a single input video $V$:

1. Source ($V_s$) and Target ($V_t$): Two distinct clips are extracted from the same video using smooth random-walk crop trajectories. These simulate independent camera movements, creating spatial misalignment and artificial occlusions between the source and target views.
2. Anchor ($V_a$): A synthetic anchor is generated by forward-warping the first frame of the source video ($V_s[0]$) to align with the target trajectory ($V_t$).
   • This uses a dense 2D tracking field (AllTracker) combined with a crop offset flow.
   • Crucial Design: The anchor is intentionally imperfect (containing holes, warping artifacts, and occlusions) to simulate the distorted inputs expected at inference time.

B. Implicit 4D Learning Mechanism

The training setup forces the model to learn implicit 4D spatiotemporal structures through a "spatial bottleneck":

• Because $V_s$ and $V_t$ are spatially misaligned, the model cannot simply copy the current source frame to fill the target.
• To reconstruct dis-occluded regions in $V_t$, the model must search the source video ($V_s$) across time to find the missing texture (e.g., finding the side of a building visible in a different frame of $V_s$).
• This "search and stitch" process compels the model to route and re-project content across different times and pseudo-viewpoints, effectively learning a 4D world prior without explicit 3D supervision.

C. Model Architecture & Conditioning

The framework adapts a pre-trained Video Diffusion Transformer (DiT) (specifically WAN 2.2):

• Token-Based Conditioning: Both the Anchor ($V_a$) and Source ($V_s$) are encoded into latents and processed as tokens within the model's self-attention mechanism (rather than using separate cross-attention layers, which proved less effective).
• Offset RoPE: To decouple the temporal position of the source condition from the target generation, a large fixed offset is applied to the Rotary Positional Embeddings (RoPE) of the source tokens.
• Auxiliary Loss: An auxiliary reconstruction loss is applied to the source tokens to ensure high-fidelity texture retention.
• Hybrid Training Strategy: While the core is trained on monocular data, a 15% mixture of synthetic multi-view data is added to help the model generalize to extreme camera rotations (e.g., 120-degree orbits) that are rare in standard monocular footage.

D. Technical Augmentations

To ensure robustness against inference-time artifacts, the training pipeline includes:

• 3D-Aware Noise: Injected into the anchor's reference frame to prevent simple texture copying.
• Fluorescent Backgrounds: High-contrast backgrounds in masked regions to provide clear boundary signals.
• Random Reference Frame Warping: Prevents bias toward early frames in the video.

3. Key Contributions

1. Scalable Self-Supervised Pipeline: A novel method to generate pseudo multi-view training triplets from abundant internet-scale monocular videos, eliminating the need for paired multi-view data.
2. Implicit 4D Routing: A mechanism that forces the model to learn 4D scene dynamics by routing missing textures across time and space to solve spatial misalignment.
3. Efficient Architecture: A minimal adaptation of pre-trained DiT models using token concatenation and Offset RoPE, achieving state-of-the-art results without heavy 3D reconstruction pipelines.
4. Hybrid Data Strategy: Demonstrating that combining real-world monocular data with a small fraction of synthetic data yields robust generalization for both texture fidelity and extreme camera control.

4. Results

The model was evaluated on a diverse set of dynamic videos against state-of-the-art methods (TrajectoryCrafter, ReCamMaster, EX-4D).

• Quantitative Performance:
  • Achieved State-of-the-Art (SOTA) performance in Temporal Consistency (99.24% Smoothness) and View Synchronization (94.96% CLIP-V).
  • Demonstrated superior Camera Accuracy (RotErr: 2.61, TransErr: 2.73) compared to most baselines, though TrajectoryCrafter had slightly lower rotation error at the cost of texture quality.
  • Significantly outperformed anchor-only methods (EX-4D) in preserving fine details (e.g., facial features, saxophone textures).
• Qualitative Performance:
  • Successfully handled complex scenarios like shaky camera footage and intricate object interactions (e.g., smoke, moving hands).
  • Avoided the "texture drift" and structural hallucinations common in anchor-only or purely synthetic models.
• Ablation Studies: Confirmed that removing the source video conditioning leads to a massive drop in texture fidelity, and that the hybrid data strategy is critical for handling extreme out-of-distribution camera trajectories.

5. Significance

Reshoot-Anything represents a paradigm shift in video reshooting:

• Democratization of Data: It proves that high-quality 4D video generation can be achieved without expensive, curated multi-view datasets, leveraging the vast abundance of standard internet videos.
• Robustness: By treating the anchor as a geometric guide rather than a texture source, the model overcomes the limitations of imperfect 3D reconstruction.
• Future Impact: The framework establishes a foundation for 4D generative world models, suggesting that complex spatiotemporal reasoning can emerge from self-supervised 2D data routing tasks. It opens new possibilities for post-production cinematography, robotics training, and interactive media.