💻 computer science

VOID: Video Object and Interaction Deletion

The paper presents VOID, a video object removal framework that leverages a vision-language model to identify and guide a video diffusion model in generating physically plausible, counterfactual outcomes for complex object interactions, addressing the limitations of existing methods in handling causal dynamics like collisions.

Original authors: Saman Motamed, William Harvey, Benjamin Klein, Luc Van Gool, Zhuoning Yuan, Ta-Ying Cheng

Published 2026-04-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Saman Motamed, William Harvey, Benjamin Klein, Luc Van Gool, Zhuoning Yuan, Ta-Ying Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a movie, and you decide you want to edit a scene. You point your finger at a character and say, "Delete them."

In the world of old video editing tools, this was like using a digital eraser. If you erased a person standing in front of a falling stack of dominoes, the tool would just fill in the empty space with the background. But here's the problem: the dominoes would still fall. The tool didn't understand that the person was holding the dominoes up. It just saw a hole to fill.

Enter VOID, a new AI model that acts less like an eraser and more like a time-traveling physicist.

The Core Problem: The "Domino Effect"

The paper explains that videos aren't just pictures moving; they are a chain of cause-and-effect.

Old AI: "I see a person. I will paint over them with the wall behind them." (Result: The person vanishes, but the ball they were holding keeps floating in mid-air, defying gravity).
VOID: "I see a person. If I remove them, the ball they were holding will drop. The dominoes they were blocking will fall. The shadow they cast will disappear." (Result: A realistic scene where physics still makes sense).

How VOID Works: The Three-Step Magic Trick

The authors built VOID using three clever tricks, which we can imagine as a team of specialists working together:

1. The "What-If" Training Camp (Data Construction)

To teach VOID how to think, the researchers didn't just show it real videos. They built a virtual simulation lab (using a tool called Kubric).

The Analogy: Imagine a video game where you can pause time, delete a character, and watch the physics engine recalculate the entire world instantly.
The Result: They created thousands of "What-If" scenarios. They showed the AI: "Here is a video of a ball hitting a wall. Now, here is the same video, but the wall is gone, and the ball flies through the air." This taught the AI that removing an object changes the future of the scene.

2. The "Super-Mask" (The Quadmask)

Usually, when you edit a video, you draw a simple black circle around the thing you want to delete.

The Old Way: "Delete the black circle."
The VOID Way: VOID uses a Quadmask, which is like a color-coded map for the AI.
- Black: The object to delete.
- Dark Grey: The object and the area where the object is touching something else (the overlap).
- Light Grey: The "Ripple Effect" zone. This is the area that will change because the object is gone (e.g., the space where a falling ball will land).
- White: Everything that stays exactly the same.
The Analogy: Instead of just telling the AI "erase this," you give it a blueprint that says, "Erase this, and also fix the mess that will happen next to it."

3. The "Brainy Assistant" (The VLM)

This is the secret sauce. The AI uses a Vision-Language Model (VLM)—basically an AI that is very good at understanding pictures and language, like a smart librarian who has read every physics textbook.

The Process: When you click on an object to delete, the VLM looks at the scene and asks, "What happens if this is gone?"
The Output: It might say, "If you remove the hand holding the balloon, the balloon will float up." It then draws the "Light Grey" map for the AI to follow. It turns a simple click into a complex plan.

The Two-Pass Editing Process

VOID doesn't just generate the video in one go; it does it in two passes to ensure perfection:

Pass 1 (The Rough Draft): It generates the new scene based on the physics plan. The objects move correctly, but they might look a little wobbly or stretched (like a claymation character).
Pass 2 (The Polish): It looks at the wobbly parts and uses "flow-warped noise" (a fancy way of saying it smooths out the motion) to make sure the objects look solid and rigid, just like they would in real life.

Why This Matters

The paper shows that VOID can handle things that other models fail at:

The Blender: If you remove the person turning on a blender, the food inside stops spinning.
The Balloon: If you remove the person holding a balloon, the balloon floats up (even though VOID was never trained on floating balloons specifically!).
The Dominoes: If you remove the middle domino, the ones after it never fall.

The Bottom Line

VOID is like giving a video editor a brain. Instead of just painting over a hole, it understands the story of the scene. It knows that if you take away the support beam, the roof falls. If you take away the brake, the car speeds up.

It bridges the gap between "making a pretty picture" and "simulating reality," proving that for video editing to truly work, the AI needs to understand the laws of physics, not just the laws of pixels.

1. Problem Statement

Existing video object removal (inpainting) methods primarily focus on photometric effects (e.g., removing shadows, reflections, or filling in the background behind an object). While effective for static or simple scenes, these models fail when the removed object has complex physical interactions with the rest of the scene.

The Core Challenge: Removing an object often triggers a chain of causal events (e.g., removing a support causes a tower to fall; removing a hand stopping a spinning top causes it to spin freely).
Current Limitations: Standard diffusion models and inpainting techniques rely on low-level visual features. They lack the high-level causal reasoning required to simulate "what would happen if this object were removed." Consequently, they often produce physically implausible results, such as objects continuing to fall after their support is removed, or failing to update the trajectory of interacting objects.

2. Methodology: The VOID Framework

VOID is a framework designed to perform physically plausible counterfactual video generation. It addresses the problem through three main axes: data construction, training strategy, and inference optimization.

A. Counterfactual Dataset Supervision

To train the model to understand physical causality, the authors generated two new paired datasets where removing an object necessitates altering downstream interactions:

Kubric (Rigid-body dynamics): ~1,900 video pairs simulating collisions, falling objects, and structural dependencies. The "counterfactual" video is generated by re-simulating the scene without the target object.
HUMOTO (Articulated interactions): ~4,500 video pairs derived from human motion capture, focusing on human-object manipulation.

Key Feature: Both datasets randomize camera trajectories and focal zooms to ensure the model learns object effects rather than camera motion artifacts.

B. Interaction-Aware Quadmask Conditioning

The authors introduce a Quadmask ( $M_q$ ) to replace the standard binary or trimask inputs used in prior work (like Generative Omnimatte). The Quadmask divides the frame into four semantic regions:

Black: The target object to be removed.
Dark Grey: The overlap between the target object and the area affected by its removal (e.g., a child catching a ball; the child is removed, but the ball's trajectory changes in that space).
Light Grey: The area affected by the removal (e.g., the ground where a ball will land, or the path of a falling object).
White: Unaffected background regions.

This explicit guidance helps the model distinguish between regions that must be synthesized (dynamic effects) and regions that must remain static.

C. Two-Pass Inference Pipeline

VOID operates in two stages to ensure both causal correctness and structural integrity:

Pass 1: Counterfactual Trajectory Synthesis
- Uses a Vision-Language Model (VLM) to analyze the input video and the user's object mask.
- The VLM identifies objects affected by the removal and predicts their new positions/trajectories in the counterfactual scenario.
- This information is converted into the Quadmask.
- The video diffusion model (based on CogVideoX) generates the initial counterfactual video ( $\hat{V}_{p1}$ ). This pass captures the correct physics (e.g., objects falling) but may suffer from structural deformation.
Pass 2: Flow-Warped Noise Stabilization
- Problem: Lightweight diffusion models often struggle with temporal coherence in complex motion, leading to "bending" or "stretching" of objects.
- Solution: If the VLM detects significant dynamic reconfiguration, a second pass is triggered. It derives warped noise from the optical flow of the first-pass output.
- This flow-aligned noise is fed back into the diffusion model to stabilize the object structure while preserving the newly synthesized motion, effectively "fixing" deformations without altering the causal outcome.

3. Key Contributions

Problem Definition: Identified and formalized the gap in current video editing models regarding physics-aware object removal and causal reasoning.
VOID Framework: A novel system integrating VLM-guided scene analysis with video diffusion models to generate counterfactual videos.
New Data & Training Strategy:
- Creation of synthetic counterfactual datasets (Kubric + HUMOTO) specifically for interaction removal.
- Introduction of the Quadmask conditioning mechanism to resolve ambiguities in interaction zones.
- A two-pass inference strategy using flow-warped noise to maintain object rigidity during dynamic changes.
Generalization: Demonstrated that the model can extrapolate to unseen physical scenarios (e.g., a balloon floating up when the holder is removed, or a blender stopping when the operator disappears) despite not being explicitly trained on those specific objects.

4. Results

The authors evaluated VOID on 75 real-world videos and 40 synthetic videos (Kubric/HUMOTO test sets), comparing against state-of-the-art inpainting models (ProPainter, DiffuEraser), effect removal models (Generative Omnimatte, ROSE), and commercial text-guided editors (Runway).

Human Preference Study: VOID was selected as the best result 64.8% of the time, significantly outperforming Runway (18.4%) and other baselines. Users cited VOID's superior ability to maintain physical plausibility and temporal consistency.
VLM-as-a-Judge Evaluation: Using Gemini 3 Pro, GPT-5.2, and Qwen 3.5, VOID achieved the highest scores across all categories, particularly in "Interaction & Physics" (3.66/5 vs. 2.61 for Runway).
Quantitative Metrics (Synthetic): On synthetic benchmarks with ground truth, VOID achieved the best FVD (260.31 vs. 437+ for others) and VLM-Judge scores, indicating superior semantic coherence and video quality.
Generalization: The model successfully handled unseen interactions, such as a Jenga tower collapsing when a hand is removed, or a ball rolling correctly when an obstacle is deleted.

5. Significance

Advancing Video Editing: VOID moves video editing beyond simple "hole filling" to world simulation. It enables non-experts to edit videos in ways that respect the laws of physics, a crucial step for high-end visual effects and automated editing.
Causal Reasoning in Generative AI: The paper demonstrates that combining VLMs (for high-level reasoning and scene understanding) with diffusion models (for pixel synthesis) is a viable path toward creating generative models that act as "simulators of the world."
Future Directions: The work highlights that current models are limited by training data diversity and resolution. It suggests that as generative models improve, the ability to perform complex causal reasoning in video editing will scale accordingly.

In summary, VOID represents a significant leap from reactive video inpainting to proactive, physics-compliant video generation, solving the "domino effect" problem where removing one element fundamentally changes the future state of the entire scene.