💻 computer science

Geometry-Instructed Video Editing

The paper introduces GIVE, a geometry-instructed video editing framework that utilizes unified object-state formulations and depth/orientation-box streams to achieve reliable, temporally coherent object-level geometric edits with consistent secondary effects like shadows and reflections.

Original authors: Chirui Chang, Xiaoyang Lyu, Yi-Hua Huang, Haoru Tan, Shizhen Zhao, Yikang Ding, Jianmin Bao, Xin Tao, Pengfei Wan, Xiaojuan Qi

Published 2026-06-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chirui Chang, Xiaoyang Lyu, Yi-Hua Huang, Haoru Tan, Shizhen Zhao, Yikang Ding, Jianmin Bao, Xin Tao, Pengfei Wan, Xiaojuan Qi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a home video of your friend walking down the street. You want to edit it, but not just by painting over pixels. You want to move your friend three steps to the left, rotate them to face the camera, or shrink them down like a toy.

In traditional video editing software (like Blender or Unreal Engine), you can do this easily because the computer knows the 3D shape of everything. But in Generative AI (the kind that creates videos from scratch), the computer usually just "guesses" what the video should look like. If you ask it to move a person, it might make them slide awkwardly, change their shadow incorrectly, or make them disappear and reappear.

This paper introduces GIVE (Geometry-Instructed Video Editing), a new way to teach AI how to perform these precise 3D moves without needing to rebuild the entire video in 3D first.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Blind" Editor

Current AI video editors are like a painter who can only see the surface of a canvas. If you tell them to "move the cup," they might smear the paint or change the cup's shape because they don't understand the cup's 3D position in space.

The Result: The edit looks messy, the shadows don't move with the object, and the video might flicker.
The Old Fix: Some methods try to build a full 3D model of the video first. But this is slow, expensive, and often breaks if the video has complex motion (like a person dancing).

2. The Solution: The "Before and After" Blueprint

GIVE doesn't try to rebuild the whole world. Instead, it uses a clever trick: it asks for a blueprint of where the object is now and where you want it to be later.

Think of it like giving a director two simple sketches:

Sketch A (The "Depth Box"): A rough outline showing where the object is sitting in the room (how far away it is and how big it is).
Sketch B (The "Orientation Box"): A simple arrow showing which way the object is facing.

You give the AI the "Before" sketch and the "After" sketch. The AI's job is simply to figure out the magic transformation that turns Sketch A into Sketch B, while keeping the rest of the video (the background, the lighting, the other people) exactly the same.

3. The Secret Sauce: The "Training Gym"

How do you teach an AI to do this perfectly? You can't just show it real videos because real videos don't have "Before and After" pairs where only one object moved.

The authors built a virtual training gym using a game engine (Unreal Engine).

The Process: They programmed the computer to create thousands of fake videos. In these videos, a robot would pick an object, move it, rotate it, or delete it, and then render the "Before" and "After" versions instantly.
The Benefit: Because it's a game engine, the computer knows exactly how the shadows should change and how the reflection should look. It creates perfect "teacher" examples for the AI to learn from.

4. How You Use It (The User Interface)

When you use GIVE, you don't need to be a 3D artist.

You upload a video.
You click on the object you want to move.
You tell the system what to do (e.g., "Rotate 90 degrees").
The system automatically uses off-the-shelf tools to guess the 3D shape and orientation, creates those "Before and After" sketches (the geometry streams), and feeds them to the AI.
The AI generates the new video.

5. What It Can Do

The paper shows that GIVE can handle a whole menu of "Digital Content Creation" (DCC) tasks:

Translation: Sliding an object to a new spot.
Rotation: Turning an object to face a different direction.
Scaling: Making an object bigger or smaller.
Duplication: Creating a copy of an object.
Removal: Making an object disappear (and filling in the background correctly).
Trajectory: Moving an object along a specific path.

The Bottom Line

GIVE is like giving the AI a pair of 3D glasses and a ruler. Instead of guessing how to move things, it looks at a simple "Before" and "After" map of the object's position and orientation. This allows it to move objects realistically, keeping shadows, reflections, and the background consistent, without needing to rebuild the entire video in 3D first.

The paper claims this method is more accurate and reliable than current commercial video editors, especially for tasks that require precise 3D movement.

Technical Summary: Geometry-Instructed Video Editing (GIVE)

1. Problem Statement

Object-level geometric editing—such as translating, rotating, scaling, duplicating, or removing objects—is a routine operation in traditional Digital Content Creation (DCC) workflows but remains unreliable in current generative video editing systems. Existing methods face two primary limitations:

Ambiguity in 3D State: Image-space or semantic controls (text prompts, masks) do not explicitly define the target object's pre- and post-edit 3D states. This leads to weak constraints where models may under-execute operations or fail to update geometry-dependent secondary effects (e.g., shadows, reflections) consistently.
Fragility of 3D Reconstruction: Methods that rely on explicit 3D reconstruction (e.g., lifting to a 3D proxy) are often costly, require per-video optimization, and are brittle under complex real-world dynamics. Inaccurate proxies propagate errors, resulting in unreliable edits and inconsistent secondary effects.

The core challenge is to specify the target object's 3D state change unambiguously across viewpoint and time while preserving non-edited regions and ensuring temporal coherence, without relying on fragile per-video reconstruction.

2. Methodology

The authors propose GIVE, a geometry-instructed video editing framework that represents edits as transitions between pre-edit and post-edit object states. The framework consists of three main components:

A. Unified Object-State Representation

Instead of using task-specific labels, GIVE defines edits through two lightweight, video-aligned geometry instruction streams derived from 3D oriented bounding boxes:

Depth-Box Stream: Encodes the coarse 3D placement and spatial extent of the object in camera coordinates. It is rasterized as a depth map (z-buffer) normalized to $[0, 1]$ .
Orientation-Box Stream: Encodes the object's heading in an appearance-agnostic manner. It renders the 3D box using fixed, canonical face-identity colors to provide anisotropic orientation cues without leaking object texture.

These streams exist for both the pre-edit (source) and post-edit (target) states. The edit operator is implicitly induced by the difference between these two states. Special cases like removal (empty post-edit streams) and duplication (multiple post-edit instances) are naturally handled within this formulation.

B. Geometry-Instructed Video Diffusion Model

GIVE instantiates the editor as a conditional latent video diffusion model (Video DiT):

Architecture: The input RGB video and the four geometry instruction streams (pre/post depth and orientation) are encoded by a shared Video VAE into spatiotemporal tokens.
Tokenization: These tokens are concatenated with the noisy edited-video latent. The sequence is processed by a Video Diffusion Transformer.
RoPE State Alignment: To ensure tokens from different streams (source vs. target) correspond to the same spatiotemporal locations, the authors align Rotary Positional Encoding (RoPE) indices across all streams. A constant state offset distinguishes source-state tokens from target-state tokens, allowing the model to infer the intended edit from the relationship between object states rather than explicit operator labels.
Inference: At inference, the system uses off-the-shelf tools (SAM 2 for segmentation, Geo4D for depth/pose estimation) to generate the geometry streams from user input, requiring no per-video optimization or engine buffers.

C. Scalable Procedural Data Synthesis

To address the lack of controlled paired supervision in real-world videos, the authors built a scalable graphics-engine pipeline (Unreal Engine) to generate GIVE-Data:

Process: The pipeline samples scenes, assets, camera trajectories, and edit programs (translation, rotation, scaling, etc.). It executes the edit program on the target object while holding all other factors (lighting, camera, background) constant.
Output: It renders controlled before/after video pairs ( $V, V'$ ) along with synchronized geometry instruction streams.
Physics: Physically based rendering ensures that geometry-dependent effects (shadows, reflections) change consistently with the transformation, providing direct supervision for faithful editing.

3. Key Contributions

Unified Object-State Formulation: A novel representation for video editing that uses pre/post geometry streams (depth and orientation boxes) to define object-state transitions, supporting diverse DCC-style operators within a single framework.
Scalable Procedural Supervision: A large-scale dataset (GIVE-Data, 202k paired clips) and benchmark (GIVE-Bench) generated via a graphics engine pipeline, isolating geometric edits and ensuring consistent secondary effects for training.
Geometry-Instructed Diffusion Model: A Video DiT architecture that integrates geometry streams via RoPE state alignment, enabling a single model to perform translation, rotation, scaling, duplication, removal, and trajectory editing.
Robust Real-World Performance: A method that achieves high edit faithfulness and temporal coherence on in-the-wild videos without requiring per-video 3D reconstruction.

4. Experimental Results

The paper evaluates GIVE on both the synthetic GIVE-Bench and 240 in-the-wild videos.

Quantitative Performance (GIVE-Bench): GIVE outperforms representative baselines (including DragVideo, Kling-O1, ProPainter, ROSE, and DaS) across all operators in PSNR, SSIM, and LPIPS. The gains are particularly significant for geometry-dependent tasks like scaling and translation, where baselines often drift or fail to update secondary effects.
Real-Video Evaluation:
- VBench Scores: GIVE achieves competitive scores in temporal consistency and video quality, often improving consistency metrics compared to baselines.
- User Study: In a blinded best-of- $K$ user study, GIVE achieved the highest WinRate across all operators (e.g., 66.75% for rotation, 76.42% for removal, 85.92% for trajectory editing), indicating stronger perceived edit faithfulness.
Ablation Studies: Removing either the orientation-box or depth-box streams degrades performance, confirming their complementary roles. RoPE state alignment is shown to be critical for maintaining spatiotemporal correspondence and improving metric scores.

5. Significance and Claims

The paper claims that GIVE addresses the central challenge of object-level video geometric revision by shifting from fragile 3D reconstruction to a unified object-state formulation driven by lightweight geometric guidance.

Faithfulness: By explicitly defining pre/post states, GIVE ensures that intended 3D changes are executed precisely, and secondary effects (shadows, reflections) update consistently with the transformation.
Scalability: The procedural data synthesis pipeline allows for the generation of large-scale, controlled training data without human annotation, enabling the learning of complex geometric transitions.
Generality: The framework supports a wide range of DCC-style operators (rotation, scaling, removal, etc.) within a single model, avoiding the need for task-specific interfaces.

The authors acknowledge a current limitation in computational efficiency, noting that concatenating multiple geometry streams increases sequence length and latency, which they identify as a direction for future work.