From Synchrony to Sequence: Exo-to-Ego Generation via Interpolation

1. Problem Definition

The paper addresses the Exo-to-Ego video generation task: synthesizing a first-person (egocentric) video from a synchronized third-person (exocentric) video and corresponding camera poses.

• Core Challenge: While synchronized data provides paired supervision, it introduces severe spatio-temporal and geometric discontinuities.
  • Visual Gap: The transition from the last frame of the exocentric view to the first frame of the egocentric view often involves a massive viewpoint shift, violating the "smooth motion" assumptions of standard video diffusion models.
  • Geometric Gap: Camera poses ($P_x$ at the end of the source sequence vs. $P_g$ at the start of the target) often exhibit abrupt jumps rather than continuous trajectories.
• Limitation of Existing Methods: Conventional "condition-output" paradigms (where the model takes the source video as a condition and outputs the target) struggle to handle these abrupt jumps, resulting in temporal incoherence and visual artifacts.

2. Methodology: Syn2Seq-Forcing

The authors propose Syn2Seq-Forcing, a framework that reframes the problem from a conditional generation task to a sequential signal modeling task.

A. Core Concept: Interpolation as Bridging

Instead of directly mapping $X \to G$, the method constructs a unified continuous sequence $S$ by interpolating between the source (Exo) and target (Ego) segments:
$$S = [X, I, G]$$
Where:

• $X$: Exocentric video frames.
• $I$: Interpolated transition frames bridging the gap between the last Exo frame and the first Ego frame.
• $G$: Target Egocentric video frames.

Similarly, the camera pose sequence is unified:
$$P = [P_x, P_i, P_g]$$
Where $P_i$ represents interpolated camera poses.

B. Key Components

1. Video Interpolation (WFLF):
   • The authors use a frozen WAN2.2 First/Last Frame (WFLF) model (a distilled variant of Wan2.2) to generate the intermediate transition frames ($I$).
   • This model takes the last Exo frame and first Ego frame as boundary conditions and generates $T-2$ intermediate frames to create a smooth visual bridge. These serve as "pseudo-ground-truth" during training.
2. Pose Interpolation:
   • Camera poses are interpolated using Spherical Linear Interpolation (SLERP) for rotation matrices and linear interpolation for translation vectors.
   • This creates a smooth geometric trajectory connecting the static (or slow-moving) exocentric camera to the dynamic egocentric camera.
3. Model Architecture (DFoT):
   • The system utilizes Diffusion Forcing Transformers (DFoT).
   • Unlike standard diffusion that assumes fixed conditioning, DFoT applies independent noise levels to each frame. This allows the model to treat the entire sequence ($X, I, G$) as a single signal where past frames condition future frames.
   • The model is trained to denoise the sequence $S^n$ conditioned on the unified pose sequence $P$.

C. Training Strategy

• Two-Stage Training:
  1. Pretraining: Trained on 356k video clips (direct Exo-Ego pairs without interpolation) to learn general video generation capabilities.
  2. Fine-tuning: Trained on category-specific data (40k clips) where the input is the full sequence $(X, I, G)$ with corresponding poses.
• Stochastic Decomposition: During training, the model randomly selects sub-tasks: either learning the transition from Exo to Interpolation ($X \to I$) or from Interpolation to Ego ($I \to G$). This forces the model to learn both the bridging transition and the final generation.

D. Inference

During inference, the model receives the clean Exo video ($X$) and the full pose sequence ($P$). It generates the transition frames ($\hat{I}$) and the final Ego frames ($\hat{G}$) in a single denoising pass, ensuring temporal coherence across the entire sequence.

3. Key Contributions

1. Problem Identification: Identified that synchronization-induced discontinuities (spatio-temporal and geometric jumps) are the primary bottleneck in Exo-to-Ego generation, rather than just conditioning quality.
2. Sequential Reformulation: Proposed Syn2Seq-Forcing, which replaces the condition-output paradigm with a sequential modeling approach using interpolation to bridge viewpoint gaps.
3. Performance Gains without Pose Interpolation: Demonstrated that interpolating only the video frames (without pose interpolation) yields significant improvements, proving that spatio-temporal discontinuity is the dominant difficulty.
4. Unified Framework: Established a flexible paradigm that naturally supports bidirectional generation (Exo2Ego and Ego2Exo) within a single continuous sequence model.

4. Experimental Results

The method was evaluated on the Ego-Exo4D benchmark across three categories: Bike, Health, and Cooking.

• Quantitative Performance:
  • Syn2Seq-Forcing outperformed state-of-the-art baselines (including TrajectoryCrafter, Wan-FCtrl, Wan VACE, and Exo2EgoSyn) across all metrics (PSNR, SSIM, LPIPS).
  • Example (Health Category): Achieved 16.71 PSNR and 0.57 SSIM, compared to ~15.6 PSNR for the next best method.
• Ablation Studies:
  • Interpolation: Removing interpolation (Direct Exo2Ego) caused a sharp drop in performance (e.g., PSNR dropped from 16.71 to 13.54 in the Health category).
  • Pose Interpolation: Adding pose interpolation provided further gains, but frame interpolation alone accounted for the majority of the improvement.
  • Pose Embedding: Plücker embeddings for camera poses yielded better results than Global vectors or Ray Encoding.
  • Interpolator Choice: Using the specialized WFLF model for interpolation significantly outperformed using the DFoT model's native interpolation capabilities.

5. Significance and Future Work

• Conceptual Shift: The paper shifts the focus from "conditioning on a jump" to "modeling a continuous trajectory," aligning video generation with the physical reality of camera motion.
• Generalizability: The framework is not limited to Exo2Ego; it offers a principled foundation for any cross-view synthesis task where viewpoint jumps exist, including Ego2Exo.
• Future Directions: The authors suggest potential improvements via higher-quality interpolation (more diffusion steps), integrated pose estimation from generated frames (using tools like ViPE), and expanding the dataset to cover more motion variability.

In summary, Syn2Seq-Forcing solves the Exo-to-Ego generation problem by explicitly smoothing the "jump" between views through interpolation, enabling diffusion models to generate high-fidelity, temporally coherent first-person videos from third-person observations.