💻 computer science

ScaleVid: Geometry-Aware Video Object Scaling with Mesh-Free Inference

ScaleVid introduces a mesh-free, two-stage training framework that achieves geometry-aware video object scaling by decoupling foreground 3D deformation from background preservation using pseudo-source reconstruction, thereby enabling realistic anisotropic resizing without explicit 3D reconstruction at inference.

Original authors: Youze Huang, Penghui Ruan, Bojia Zi, Xianbiao Qi, Shihao Zhao, Rong Xiao

Published 2026-08-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Youze Huang, Penghui Ruan, Bojia Zi, Xianbiao Qi, Shihao Zhao, Rong Xiao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are holding a magic remote control for reality. You point it at a video of a cat, press a button, and suddenly the cat is twice as wide, or the car in the background shrinks to the size of a toy. This is the dream of "video editing" powered by artificial intelligence. For a long time, computers were great at changing the look of things—making a shirt red or removing a person from a photo. But changing the shape or size of an object in a video while keeping it looking real? That's like trying to stretch a rubber band without it snapping or turning into a weird blob. The computer has to understand that if you stretch a car sideways, its wheels shouldn't just stretch like taffy; they should look like they are actually getting wider in 3D space, and the background behind it needs to stay perfectly still. If the computer gets this wrong, the video looks like a glitchy nightmare where objects melt or float in impossible ways.

This is where a new idea called "ScaleVid" comes in. The researchers behind it wanted to solve a specific puzzle: how to resize objects in a video (making them bigger, smaller, wider, or taller) without needing to build a complex 3D model of the world first. Usually, to do this kind of resizing, you'd need to scan the object, build a 3D mesh (like a wireframe skeleton), stretch that skeleton, and then paint the video back onto it. It's slow, expensive, and requires a lot of manual work. ScaleVid suggests a smarter, faster way. Instead of building a 3D model, it uses a two-step "training" trick. First, it teaches a computer to stretch things in simple 2D ways (like squashing a photo flat). Then, it teaches the computer to understand how those stretches look when they happen in 3D space, using fake "practice" videos made from 3D models. Once trained, the computer can take a real video, stretch the object exactly how you want, and fill in the background perfectly, all without ever building a 3D model during the actual editing process.

The paper, titled "ScaleVid: Geometry-Aware Video Object Scaling with Mesh-Free Inference," presents a method that achieves this by splitting the job into two distinct stages. The authors argue that existing methods either rely on text prompts (which are often imprecise, like telling a computer to "make it big" without saying how big) or require heavy 3D reconstruction (which is too slow for real-time use). ScaleVid proposes a "progressive" approach. In the first stage, the system learns to handle simple, flat (2D) changes, like stretching a rectangle. This teaches it how to keep the background stable while the object changes. In the second stage, it learns to handle true 3D deformations. Here, the system uses a special "Deformer" module that takes a video and applies a specific 3D stretch based on user instructions (like "stretch width by 1.5x, height by 0.8x"). Crucially, the system is trained using "pseudo-sources"—fake videos created by stretching 3D models and then rendering them back into video format. This allows the AI to learn the rules of 3D geometry without needing real-world videos where the object is already resized (which are almost impossible to find).

The results are quite impressive. When tested on a variety of objects, from transparent glass goblets to complex cars, ScaleVid managed to resize them while keeping the geometry looking correct. For instance, if you stretch a car's depth, the rear wheels become visible in a way that makes sense physically, something simple 2D stretching cannot do. The paper shows that ScaleVid outperforms other methods in keeping the object's identity (it still looks like the same car, just bigger) and preserving the background (no weird warping of the sky or road). The authors measured this using several metrics, finding that their method achieved a scale accuracy score of 0.804 (IoU) and a geometric alignment error of only 0.237 degrees for yaw, significantly better than competitors like "DiffHandles" or "GeoDiffuser."

However, the paper is careful to note that this isn't a magic wand that fixes everything. The system still has limitations. If the object is very symmetrical, like a perfect sphere or a cylinder, the computer might get confused about which way is "width" and which is "depth," leading to ambiguous results. Also, if the object shrinks significantly, the system has to "invent" the background that was previously hidden behind the object, and sometimes it might hallucinate weird details in those new areas. The authors suggest that while their method is a major step forward, future work could focus on better handling these "invention" tasks and making the 3D stretching even more robust for tricky shapes.

In essence, ScaleVid is like a master chef who learns to cook a complex dish by first practicing on a simple, fake version of the ingredients, and then applying those skills to real, fresh food. By separating the "stretching" logic from the "painting" logic, the researchers created a system that can resize objects in videos with a level of geometric realism that was previously out of reach for fast, mesh-free methods. It suggests that we don't always need to build a full 3D world to edit it; sometimes, teaching the AI to understand the rules of 3D space through clever training is enough to make the magic happen.

Technical Summary: ScaleVid

Problem Definition

The paper addresses the challenge of geometry-aware video object scaling, a task that requires anisotropically resizing a target object along its object-centric axes (width, height, and depth) while maintaining geometric plausibility, temporal coherence, and background consistency.

Existing approaches face significant limitations:

Text-guided methods operate primarily in the 2D image plane, lacking precise control over 3D geometric transformations.
Depth-guided approaches offer only coarse geometric control.
Mesh-based methods require costly 3D reconstruction, camera estimation, and explicit mesh-pixel alignment at inference, which hinders scalability and practical application.
Data Scarcity: There is a lack of paired real-world training videos where an object is scaled in 3D while the background remains consistent, making supervised learning difficult.

The goal is to scale an object according to user-specified 3D factors ( $s_x, s_y, s_z$ ) without resorting to explicit 3D reconstruction or camera estimation during inference.

Methodology

ScaleVid introduces a progressive two-stage training framework that decouples geometric transformation from realistic video synthesis using a pseudo-source reconstruction strategy. The core insight is to construct geometrically perturbed pseudo-sources from real videos while retaining the original complete videos as reconstruction targets. This allows the model to learn geometric control from synthetic supervision and realistic composition from real-video data without explicit mesh-pixel alignment.

1. Geometry-Aware Guidance Modules (Deformer & Masker)

These modules are trained on a large-scale synthetic dataset constructed from 3D meshes (reconstructed via Hunyuan3D) and rendered with controlled anisotropic scaling.

Canonical Axis Alignment: To ensure consistent scaling semantics across different objects, the method aligns the object's Oriented Bounding Box (OBB) axes with a renderer's canonical frame (upright $y$ , width $x$ , depth $z$ ).
Deformer: A latent-space conditional flow model that learns to transform the foreground video based on continuous anisotropic scaling parameters. It employs a bidirectional training objective (forward and inverse scaling) to ensure transformation consistency and reduce spatial drift.
Masker: Predicts the temporally consistent spatial support (mask) of the transformed foreground. It is distilled into a 3-step generator for efficiency.

2. Progressive Training of the Main Model

The Main Model is trained in two stages to progressively learn from 2D to 3D transformations:

Stage I (2D-Planar Scaling): The model learns robust foreground-background composition using planar transformations. Pseudo-sources are constructed by applying 2D scaling to the foreground and masking the corresponding region in the background. This stage establishes a strong prior for completing missing regions and preserving the scene.
Stage II (3D-Aware Scaling): The model incorporates object-centric 3D deformation guidance. The Deformer generates a geometrically transformed foreground ( $F_{src}$ ), and the Masker predicts its support. Crucially, the Main Model is conditioned on a target-aligned guidance ( $F_{tgt}$ ) generated by applying the inverse scaling to $F_{src}$ . This creates a non-trivial learning signal where the Main Model must refine the deformed foreground and seamlessly compose it with the background, rather than simply copying the input.

3. Inference Pipeline

At inference, the process is streamlined and mesh-free:

The user selects an object (via SAM2) and specifies scaling factors $s$ .
The background is extracted (using Minimax Remover).
The Deformer is applied once to generate the target-aligned foreground.
The Main Model integrates this foreground with the background to produce the final video.
The Masker and inverse deformation are not required at inference, significantly reducing computational cost.

Key Contributions

Formulation of Video Object Scaling: The paper formulates the task using object-centric anisotropic factors and learns projected geometric effects without explicit 3D reconstruction at inference.
Progressive 2D-to-3D Training Strategy: A novel framework that constructs pseudo-sources while using original real videos as targets. This enables learning robust composition from planar transformations (Stage I) and geometry-aware scaling from 3D deformation guidance (Stage II).
Comprehensive Benchmarks: The authors establish three complementary benchmarks:
- Geometry Benchmark: Isolates geometric accuracy and foreground fidelity on uniform backgrounds.
- Real-Background Benchmark: Composites rendered objects into real-world backgrounds to test background preservation.
- Real-World Benchmark: Evaluates performance on in-the-wild videos (Pexels, DAVIS) focusing on perceptual quality and temporal consistency.

Experimental Results

Extensive experiments demonstrate that ScaleVid outperforms state-of-the-art methods (including text-guided, depth-guided, and mesh-based approaches) across multiple metrics:

Geometric Accuracy: ScaleVid achieves superior scale accuracy and geometric alignment (measured by mask IoU, area error, and camera pose fitting errors) compared to methods relying on text prompts or depth maps.
Foreground Fidelity: The method yields the best results in pixel-level error (MSE, PSNR) and perceptual similarity (DINO, DreamSim) on both synthetic and real-background benchmarks.
Background Preservation: It maintains high background consistency, outperforming methods that struggle with object-scene interactions or boundary leakage.
Temporal Consistency: ScaleVid achieves competitive Ewarp scores, indicating stable temporal coherence without the flickering seen in image-based or text-driven methods.
Efficiency: By avoiding explicit 3D reconstruction and mesh rendering at inference, ScaleVid offers faster and more practical inference compared to mesh-based pipelines.

Significance and Claims

The paper claims that ScaleVid represents a significant step forward in controllable video editing by successfully decoupling geometric transformation from high-fidelity video synthesis. Its significance lies in:

Eliminating the Inference Bottleneck: It achieves geometry-aware scaling without the computational overhead of 3D reconstruction or camera estimation during the generation phase.
Solving the Data Scarcity Problem: The pseudo-source reconstruction formulation allows the model to learn from real videos without requiring paired real-world scaling targets, which are otherwise unavailable.
Practical Applicability: The method is suitable for short-video editing, advertising, and AR/VR applications where object size affects composition and visual emphasis, offering a balance between geometric control and realistic appearance that existing methods fail to achieve simultaneously.

The authors acknowledge limitations, particularly regarding the distribution gap between mesh-rendered training data and real pixel-space videos, and the ambiguity of canonical axes for symmetric objects (e.g., spheres). However, they assert that the proposed framework provides a robust and practical solution for geometry-aware video object scaling.