💻 computer science

Neural Voxel Dynamics: Learning Implicit 3D Physics via Volumetric Feature Advection

This paper introduces a self-supervised framework that learns implicit 3D physical dynamics by unprojecting video features into a volumetric latent space and modeling them as action-conditioned advection, enabling the emergence of physically consistent, long-term 3D world models from monocular videos without relying on explicit physics simulators.

Original authors: Zican Wang, Niloy Mitra

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zican Wang, Niloy Mitra

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Teaching AI to "Feel" Physics

Imagine you are watching a video of a glass of water being knocked over. A standard AI video generator might look at the pixels and guess, "Okay, the water looks like it's splashing, so I'll make the next frame look like a splash." It's good at copying the look, but it doesn't actually understand why the water splashes or how it will behave if you push it again. It often fails to keep the glass intact or might make the water float away like magic.

This paper introduces a new system called Neural Voxel Dynamics. Instead of just guessing what pixels should look like next, this system tries to build a hidden, 3D "mental model" of the world inside the computer. It learns the actual rules of physics (like gravity, collisions, and fluid flow) just by watching videos, without needing a human to teach it the math formulas.

The Problem: The "Flat" vs. "Deep" View

Current video AI models work like a painter looking at a flat canvas (2D). They are great at making pretty pictures, but they don't understand that a ball rolling behind a tree is still there; they just think the ball disappeared. They lack object permanence and physical consistency.

The authors argue that to truly understand physics, the AI needs to stop thinking in 2D images and start thinking in 3D space.

The Solution: The "Invisible Lego" Box

The researchers built a system that turns a flat video into a 3D structure made of invisible blocks called voxels (think of them as 3D pixels, like tiny Lego bricks filling a room).

Here is how their system works, step-by-step:

1. Lifting the Video into 3D (The "Un-Flatting" Machine)

The system takes a regular video (which is just a flat picture changing over time) and uses a "depth guesser" to figure out how far away things are. It then projects the video's features into a 3D grid of these invisible Lego blocks.

Analogy: Imagine taking a flat shadow puppet show and suddenly realizing the puppets are actually 3D marionettes hanging in a room. The system reconstructs the 3D room from the 2D shadow.

2. The "Feature Advection" (The Invisible Wind)

Once the world is built as a grid of 3D blocks, the system doesn't simulate physics with heavy math equations (like calculating friction or viscosity). Instead, it treats physics like smoke moving through the air.

The Analogy: Imagine the 3D blocks are filled with invisible "information smoke." When you push a block (apply a force), the system learns how that "smoke" drifts and swirls to the next block.
If you push a rigid block (like a cube), the "smoke" moves as a solid chunk.
If you push a fluid (like water), the "smoke" spreads out and splashes.
The AI learns these patterns automatically. It doesn't need to be told "this is water" or "this is a rock." It just learns that this type of information flow happens when this type of force is applied.

3. Learning from "Ghost" Observations

Since the AI only sees the video from one camera angle, it can't see the back of objects. The system is smart enough to guess what's happening in the "unseen" blocks.

Analogy: If you see a ball roll behind a wall, a human knows the ball is still there, just hidden. This AI does the same thing. It keeps the "ghost" of the ball alive in the hidden 3D blocks so that when the ball comes out the other side, it behaves correctly.

Why This is a Big Deal

Most other methods try to mix AI with traditional physics engines (like those used in video games). But those engines need humans to manually set up rules: "This is a solid," "This is liquid," "This is heavy."

This new method is self-supervised. It learns everything on its own just by watching videos.

It can handle rigid bodies (rocks), fluids (water), and smoke all in the same system without switching gears.
It creates a "World Model" that understands cause and effect. If you push a cup, it knows the cup will fall, even if the video cuts off before it hits the ground.

The Results

The researchers tested their system on several benchmarks (like CLEVRER and PhysInOne) involving bouncing balls, pouring liquids, and smoke.

The Outcome: Their model predicted future movements much more accurately than previous AI models. It kept objects solid, made fluids flow naturally, and didn't make objects disappear or teleport.
The Proof: Even when they tested it with just one camera view (which is harder), it still managed to understand the 3D physics better than models that only looked at 2D pixels.

Summary

In short, this paper teaches an AI to stop just "painting" the next frame of a video and start "simulating" the 3D world inside a hidden grid. By treating physics as a flow of information through 3D blocks, it learns to predict how the real world moves, crashes, and flows, all without needing a human to write the physics textbook for it.

Technical Summary: Neural Voxel Dynamics

Problem Statement
Current generative video models, while achieving high visual fidelity, often lack a 3D geometric foundation, leading to physical inconsistencies and failures in maintaining object permanence. These models typically operate in 2D latent spaces, "morphing" pixels through learned shortcuts rather than simulating underlying 3D physical transformations. Conversely, explicit physics-based methods (e.g., those using Material Point Method or Position Based Dynamics) are hindered by the "simulation gap"—the difficulty of estimating precise physical parameters from raw pixels—and require rigid pre-specification of material types (e.g., solid vs. fluid) and manual parameter tuning. There is a need for a framework that can implicitly learn 3D physical dynamics directly from passive monocular video observations without relying on explicit physics engines or ground-truth physical labels.

Methodology: Neural Voxel Dynamics
The proposed framework, Neural Voxel Dynamics, shifts the predictive bottleneck from 2D image space to a "lifted" 3D Volumetric Latent Space. The method operates in two primary stages:

Geometric Lifting to Voxel Latents:
- The system takes monocular video frames and semantic features extracted from a Video Joint-Embedding Predictive Architecture (V-JEPA).
- Using monocular depth priors (specifically MoGe), these 2D semantic features are unprojected into a canonical 3D voxel grid.
- This process creates a sparse 4D spatial-temporal voxel representation ( $Z_t$ ) that includes three channels: semantic features ( $H_t$ ), an occupancy probability channel ( $O_t$ ), and an observed/unobserved channel ( $U_t$ ). This structure enforces object permanence and occlusion reasoning, resolving spatial ambiguities inherent in 2D representations.
Implicit Feature Advection:
- The core predictor is a Volumetric Feature Advection model, formulated as an action-conditioned generative transition operator.
- Instead of using a classical solver, the model learns a time-dependent velocity field via flow matching to transport noisy source distributions toward the target latent manifold.
- The architecture utilizes a Dynamic Local Spatiotemporal Transformer (a Diffusion Transformer variant). It employs:
  - Local Spatial Attention: Voxels attend only to a sparse $k \times k \times k$ neighborhood, mimicking local force propagation in physical systems and ensuring linear computational scaling.
  - Temporal Attention: Tokens are regrouped by spatial location to track trajectories over time, aggregating momentum and velocity cues.
- Conditioning: The transition is conditioned on external forces represented as a 12-dimensional vector (encoding contact point, direction, magnitude, duration, etc.), integrated via local element-wise addition and global modulation tokens.

Key Contributions

Lifting Semantic Features: A method to unproject semantic V-JEPA features into a grounded, sparse 3D latent voxel volume using monocular depth, creating a representation that is both geometrically grounded and materially implicit.
Volumetric Feature Advection: An action-conditioned architecture that learns implicit 3D dynamics as a spatio-temporal state advection problem, unifying the simulation of heterogeneous phenomena (rigid bodies, fluids, smoke) within a single pipeline without pre-specifying material types.
End-to-End Learning: A framework supervised solely by video-derived signals and action conditions, requiring no access to physics engine internal states, labels, or surrogate models.

Experimental Results
The model was evaluated on multiple benchmarks, including CLEVRER (rigid bodies), PhysInOne (complex objects and fluids), and PhysGaia (fluids and smoke).

Latent Prediction: The model outperformed state-of-the-art 2D world models (e.g., CogVideoX) and explicit hybrid methods (e.g., PhysGen, PhysGaussian, PhysCtrl) in terms of 2D and 3D latent L2 loss. It demonstrated superior consistency across multi-camera views, whereas 2D baselines degraded significantly when evaluated from multiple angles.
Material Generalization: Unlike explicit simulators that struggle with mixed material classes, the proposed method successfully simulated interactions between rigid bodies, fluids, and smoke without explicit material segmentation.
Flow Estimation: The predicted latent dynamics exhibited lower flow estimation errors compared to baselines, indicating that the learned features capture physically plausible motion patterns consistent with ground truth.
Ablations: Experiments confirmed the necessity of the 3D projection, the projection loss (which improves geometric overlap), and the inclusion of occupancy/observed channels for accurate geometry learning.

Significance and Claims
The authors claim that this work opens a scalable pathway toward general-purpose dynamic world models that internalize the 3D invariants of the physical world solely through passive observation. By treating physics as a spatio-temporal state advection problem in a high-dimensional latent space, the approach bridges the gap between high-level semantic features and physical invariants. The significance lies in its ability to generate physically grounded, perspective-independent forecasts and simulate complex, heterogeneous interactions without manual preprocessing, segmentation, or reliance on explicit physics engines. The authors position this as a step toward "true dynamic world models" that possess an intuitive understanding of physical laws derived from video data.