🤖 AI

Geometric 4D Stitching for Grounded 4D Generation

This paper proposes Geometric 4D Stitching, an efficient framework that explicitly identifies and fills missing geometric regions with grounded 4D stitches to resolve inconsistencies in existing 4D generation methods, enabling fast, high-quality scene expansion and editing on a single GPU.

Original authors: Sunwoo Park, Taesung Kwon, Jong Chul Ye

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sunwoo Park, Taesung Kwon, Jong Chul Ye

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a 3D model of a room, but you only have a few blurry photos taken from one corner. You want to see what the room looks like from the other side, but you've never been there.

Current methods try to solve this by asking a "magic artist" (a generative AI) to draw every single missing angle of the room. The problem is, this artist is great at making things look pretty, but they often mess up the structure. They might draw a chair that looks real but is floating in mid-air, or a wall that bends in impossible ways. When you try to stitch all these drawings together into a 3D model, the model becomes wobbly and inconsistent because the "artist" didn't follow the rules of physics or geometry.

Geometric 4D Stitching (or G4S) is a new method that changes the game. Instead of asking the artist to redraw the whole room, it acts like a smart construction foreman. Here is how it works, using simple analogies:

1. The Anchor: "The Solid Foundation"

First, the method takes the video you already have (the "source view") and builds a rough, solid 3D skeleton of what it knows is true. Think of this as the foundation of a house. It's not perfect, but it's grounded in reality because it comes from actual observations.

2. The Detective Work: "Finding the Gaps"

Next, the method asks the AI to imagine what the room looks like from a new angle. But instead of trusting the AI to draw the entire new picture, G4S acts like a detective. It compares the AI's drawing with the solid foundation it already built.

The "Curtain" Problem: Sometimes, when you move your head, you see things hidden behind a sofa. In 3D modeling, the AI often tries to "connect the dots" across these hidden spots, creating thin, invisible sheets of geometry called "curtains" that don't actually exist.
The Solution: G4S identifies exactly where the AI is guessing (the gaps and the "curtains") and marks those specific spots as "Information Addition Regions." It ignores the parts where the AI just copied what was already known.

3. The Stitch: "Sewing in the New Fabric"

This is the core innovation. G4S doesn't paste the AI's entire drawing onto the model. Instead, it takes only the new, hidden parts the AI generated (the "newly revealed regions").

Before sewing these new pieces in, it "irones" them out. It forces the new geometry to align perfectly with the solid foundation (the anchor) so that walls stay straight and objects don't float.
It then "stitches" these reliable, corrected pieces into the 3D model. If a piece of the AI's drawing looks suspicious or conflicts with the foundation, it gets thrown away.

4. The Result: "A Coherent 4D World"

The final result is a 4D scene (a 3D world that moves over time) that is:

Geometrically Grounded: It follows the rules of physics because it's anchored to real observations.
Fast: It doesn't need to spend hours tweaking the model. It builds the scene in under 10 minutes on a powerful computer.
Editable: Because it's built on a solid mesh (like a wireframe) rather than a blurry cloud of light, you can actually move objects around or remove them later without the whole thing falling apart.

Why is this better than the old way?

Think of the old method (Radiance-based reconstruction) like trying to build a sculpture by pouring wet clay and hoping it dries into the right shape. If the clay is inconsistent, the sculpture looks okay from one angle but collapses from another.

Geometric 4D Stitching is like building with LEGO bricks. You start with a solid base. When you need to add a new section, you only build the specific bricks that are missing, double-check that they snap perfectly into the existing structure, and then lock them in. If a brick doesn't fit, you don't force it; you just don't use it.

In short: This paper proposes a way to expand 3D video worlds by only filling in the missing pieces with "grounded" geometry, ensuring the final result is stable, consistent, and easy to edit, all while avoiding the expensive and error-prone process of trying to reconstruct the entire scene from scratch.

Technical Summary: Geometric 4D Stitching for Grounded 4D Generation

Problem Statement

Recent advances in 4D generation rely on synthesizing multi-view videos using generative models (e.g., Video Diffusion Models) and reconstructing them into radiance-based representations like Neural Radiance Fields (NeRF) or 3D Gaussian Splatting (4DGS). However, this pipeline faces two critical limitations:

Geometric Inconsistency: Generative models prioritize visual plausibility over strict 3D structural consistency. When synthesizing novel views, they often produce geometrically misaligned content. In sparse-view settings, these inconsistencies amplify, creating an ill-posed reconstruction problem where conflicting geometric evidence prevents the formation of a coherent 3D scene.
Inefficiency and View-Dependent Artifacts: Radiance-based reconstruction attempts to absorb these geometric inconsistencies into view-dependent effects (e.g., opacity or spherical harmonics) rather than resolving the underlying geometry. This leads to artifacts like "inpainting collapse" and limits downstream manipulability. Furthermore, achieving dense multi-view consistency through optimization is computationally expensive and often fails to converge to a stable solution under sparse observations.

Methodology: Geometric 4D Stitching (G4S)

The authors propose Geometric 4D Stitching, a framework that reframes 4D generation from dense scene-level completion to explicit, region-level geometric augmentation. Instead of optimizing a radiance field over all views, G4S constructs an explicit 4D mesh by selectively stitching geometrically grounded "stitches" into an initial asset.

The pipeline consists of four main stages:

1. Initial-View 4D Asset Generation

Given a source video (observed or generated), the method estimates camera poses and depth using a feed-forward geometry transformer (DA3). A triangular mesh is constructed on the image lattice by connecting neighboring valid pixels. Vertices are lifted into 3D space using the estimated depth and camera parameters, forming a time-indexed raw mesh sequence. Background completion is performed at this stage to mitigate occlusion-induced information gaps.

2. Novel View Synthesis with Information-Addition Regions

Rather than synthesizing entire novel views, the method identifies specific regions where new information is genuinely required.

Projection Holes: Pixels in the target view that cannot be projected from the source view are identified as missing.
Curtain Detection: The method compares renderings of the raw mesh and a point-cloud representation of the same source. Discrepancies (specifically "curtains" formed by incorrect mesh connections across occlusion boundaries) reveal occlusion-induced missing regions.
Candidate Generation: A Novel View Synthesis (NVS) module generates content only for these identified "information-addition regions," creating candidate stitch patches rather than full-frame novel views.

3. Geometric Alignment via Pyramidal Refinement

To ensure the generated content aligns with the existing asset, the method refines the depth of the novel-view candidates.

Local Scale-Shift Field: Instead of global correction, a local scale-shift field $\Theta(x) = (s(x), b(x))$ is estimated patch-wise to correct depth misalignments.
Pyramidal Optimization: The refinement proceeds from coarse to fine scales. The objective function minimizes the difference between the refined depth and an "anchor" depth (rendered from the existing asset) in valid regions, while enforcing geometry-aware smoothness and stability in non-anchor regions. This ensures the new geometry is anchored to the reliable source view.

4. Geometric 4D Stitching and Visual Refinement

Selective Stitching: Only the geometry-refined candidate regions are back-projected and inserted into the initial 4D mesh. This avoids contaminating reliable existing geometry with potentially inconsistent NVS outputs.
Render-Disagreement Distillation: To further filter unstable additions, the scene is rendered back to observed views. Candidates that consistently disagree with the anchor render are removed.
Visual Refinement: A lightweight, single-step diffusion-based refinement (using TrajectoryCrafter) is applied to the rendered mesh to reduce high-frequency boundary artifacts without altering the underlying geometry.

Key Contributions

Identification of a Fundamental Limitation: The paper highlights that increasing observation density for 4D generation often exacerbates multi-view consistency costs under current NVS model capacities, leading to geometric collapse.
Explicit Geometric Stitching Formulation: The authors reformulate 4D generation as an explicit geometric stitching problem. By integrating geometrically aligned stitches only into incomplete regions, the method avoids redundant view generation and minimizes alignment costs.
Efficiency and Grounded Consistency: The method constructs grounded 4D scene representations in under 10 minutes on a single NVIDIA RTX 5090 GPU, significantly faster than per-scene radiance optimization, while enforcing strict geometric consistency.
Practical Downstream Applications: The explicit mesh representation enables tasks difficult for radiance-based methods, such as geometrically aligned scene extrapolation and 4D scene editing (object removal/addition).

Experimental Results

The authors evaluate G4S against radiance-based baselines (D-NeRF and 4DGS) using sparse-view inputs generated by SORA2.

Perceptual Quality: Using VBench, G4S achieves higher scores in Subject Consistency, Background Consistency, and Image Quality compared to baselines, indicating more stable and visually plausible novel views.
Geometric Robustness: Camera pose estimation (using DA3) on rendered videos shows significantly lower Absolute Trajectory Error (ATE) and rotation errors for G4S. This confirms that the reconstructed 4D scene maintains better spatial consistency with the intended camera trajectory.
Self-Consistency: Reprojection evaluations demonstrate that G4S produces spatially coherent novel views with reduced cross-view discrepancies, whereas radiance-based methods often exhibit view-dependent flickering and structural distortion.

Significance and Claims

The paper claims that Geometric 4D Stitching offers a viable alternative to the conventional paradigm of dense generative observation followed by radiance-field optimization. By explicitly identifying missing regions and grounding them in existing geometry, the method bypasses the ill-posed nature of generative 4D reconstruction under sparse views.

The authors emphasize that their approach does not require per-scene optimization, making it efficient and scalable. They position the work as a step toward building coherent 4D assets from sparse observations without relying on the "black box" of dense generative completion. However, they modestly note that performance is still bounded by the capabilities of the underlying NVS backbone and feed-forward geometry estimators, and that generated unrevealed regions should be treated as creative or simulation tools rather than metrically accurate real-world reconstructions.