Technical Summary: TeleMorpher

Problem Statement

While diffusion models have achieved significant success in video generation and editing, the simultaneous editing of both motion and location remains a largely unexplored and challenging frontier. Existing motion editing methods (e.g., MotionEditor, Edit-Your-Motion) primarily focus on one-shot editing but often fail to address location changes, which are critical for practical applications. These existing approaches suffer from several fundamental issues:

• Appearance Inconsistency: Flickering and drift in the protagonist's appearance or background.
• Temporal Inconsistency: Lack of smooth transitions across frames.
• Motion-Location Gap: Difficulty in bridging the large discrepancy between source and target motions, especially when the target involves significant spatial shifts (location changes).
• Dependency on References: Reliance on real-world reference videos limits the variety and controllability of target motions.

The authors identify that the degradation in editing quality stems from three main factors: Gap (differences in motion and location between source and target), Ambiguity (low resolution or high similarity between protagonist and background), and Information Amount (complex backgrounds or rapid motion dynamics).

Methodology: TeleMorpher

TeleMorpher is a one-shot, training-free framework designed for robust simultaneous motion-location editing. It operates through a modular pipeline that disentangles the editing process to address the identified challenges:

1. Foreground-Background Disentanglement

To eliminate background interference and focus the model on the protagonist, the pipeline first separates the input video ($X_s$) into:

• Protagonist Video ($X_p$): Extracted using a pre-trained segmentation model (Segment Anything).
• Background Video ($X_b$): The remaining background is filled using an inpainting model (Inpaint Anything) to create a clean background for later reintegration.

2. Motion Prior Utilization

Instead of relying on real-world reference videos, TeleMorpher utilizes synthetic motion priors ($X_m$) generated from highly controllable 3D avatar systems. These priors serve as the ground truth for the target motion, offering:

• Greater controllability and scalability.
• Elimination of the need for manual rendering or additional avatar controls.
• Access to a wider variety of motion patterns than existing video datasets.

3. Training-Free Pose Warping

To bridge the gap between the source motion and the target motion prior while preserving the protagonist's appearance, the framework introduces a training-free pose warping strategy. This acts as a residual-style guidance mechanism:

• Mask Analysis: It computes the intersection and symmetric difference between source and target segmentation masks to identify structural discrepancies (e.g., extended limbs).
• Skeleton Alignment: It extracts skeletons from both source and target to isolate corresponding body parts (e.g., left arm to left arm).
• Warping Strategies:
  • Parametric Warping: Fits quadratic curves to the centerlines of corresponding body parts to calculate vertical displacement ($\Delta y$) and warp the source image pixels.
  • Stochastic Synthesis: For cases where geometric fitting fails (due to occlusion or fragmentation), it randomly samples and redistributes pixels from the source to the target shape to maintain local appearance.

4. Protagonist-Guided Inference

The warped protagonist condition ($P$) is injected directly into the UNet's attention layers of a baseline motion editor (MotionEditor) during inference:

• Attention Injection: A subset of the attention keys ($K$) and values ($V$) are overwritten with repeated RGB features from the protagonist condition to enforce appearance and motion consistency.
• Mixup Strategy: The input video ($X$) is linearly mixed with the protagonist condition ($P$) using a coefficient $\lambda$ to further reinforce the target conditioning effect.

5. Final Integration

The edited protagonist video is merged with the inpainted background video to produce the final output, ensuring seamless blending.

Key Contributions

1. Problem Analysis: The authors provide a systematic analysis of the factors degrading simultaneous motion-location editing, categorizing them into gap, ambiguity, and information amount.
2. TeleMorpher Pipeline: The introduction of a one-shot framework that leverages motion priors (synthetic 3D avatar motions) instead of real reference videos, enabling more precise and flexible control.
3. Training-Free Guidance: A novel pose warping method that aligns source and target motions without additional network training, reducing the gap while preserving appearance.
4. New Evaluation Metrics: The proposal of two new LPIPS-based metrics:
   • LPIPS-B (Background): Measures background preservation by isolating background regions.
   • LPIPS-P (Pose): Measures motion alignment by comparing extracted skeletons between the edited output and the target prior.
5. Resource Release: The public release of the code, collected videos, and baseline results (TeleMorpher-smallpack).

Experimental Results

The framework was evaluated on 20 in-the-wild videos (YouTube, Hailuo AI, Pexels) and the TaiChi dataset, comparing against state-of-the-art baselines (Follow-Your-Pose, ControlVideo, MasaCtrl, MotionDirector, MotionEditor).

• Quantitative Performance: TeleMorpher achieved the best scores in LPIPS-S (Source appearance preservation), LPIPS-B (Background preservation), LPIPS-P (Pose alignment), and CLIP Score (Semantic alignment). It demonstrated superior temporal consistency (LPIPS-N) and target motion adherence (LPIPS-T) compared to most baselines.
• Qualitative Performance: Visual comparisons showed that TeleMorpher successfully edited motion and location simultaneously while maintaining the protagonist's appearance and background integrity, avoiding the flickering and appearance drift seen in other methods.
• User Study: In a study with 25 participants, TeleMorpher was preferred over all baselines in terms of Motion Alignment (M-A), Appearance Alignment (A-A), and Textual Alignment (T-A).

Significance and Claims

The paper claims that TeleMorpher bridges the gap between the high controllability of modern 3D avatar platforms and the editing of real-world human videos. By leveraging motion priors and a disentangled editing pipeline, it allows users to specify desired motions through virtual avatars and transfer them to existing videos without additional rendering, animation, or manual control.

The authors position TeleMorpher as a step toward scalable, precise editing of video structures, laying the groundwork for future applications in object editing, protagonist-object interaction, and multi-protagonist scenarios. The work underscores that separating the protagonist from the background and utilizing synthetic motion guidance are critical for achieving robust, high-fidelity simultaneous motion-location editing.

Limitations

The authors acknowledge two primary limitations:

1. Color Shifts: Occasional color inconsistencies in the edited protagonist, which could be mitigated by post-processing color correction.
2. Video Length: Experiments were limited to short clips (5 frames) due to computational resource constraints. The authors suggest iterative application on segmented clips as a potential mitigation strategy.