PhysConvex: Physics-Informed 3D Dynamic Convex Radiance Fields for Reconstruction and Simulation

Imagine you are trying to teach a computer to understand a squishy, wobbly toy (like a stress ball or a jellyfish) just by watching a video of it bouncing around.

Current computer vision tools are great at taking a photo and making a pretty 3D model that looks real. But if you try to make that model move, it often falls apart. It might stretch like silly putty when it should snap back like a rubber band, or it might pass through walls because the computer doesn't understand the laws of physics.

PhysConvex is a new invention that solves this by combining "seeing" with "feeling." Here is how it works, using some everyday analogies:

1. The Old Way: The "Fuzzy Cloud" vs. The "Rigid Box"

Most previous methods tried to build 3D objects using tiny, fuzzy clouds (like 3D Gaussians) or rigid blocks (like voxels).

The Problem: Imagine trying to model a wobbly Jell-O mold using only rigid Lego bricks. You can get the general shape, but you can't make it jiggle or stretch naturally. Or, imagine trying to model a sharp corner using only soft, fuzzy clouds; the edges always look blurry and mushy.
The Limitation: These tools are great at rendering (making it look pretty) but terrible at simulating (making it move correctly). They don't know that if you push a rubber ball, it squishes in a specific way based on its material.

2. The PhysConvex Solution: The "Smart Polyhedron"

PhysConvex changes the game by using Dynamic Convex Primitives.

The Analogy: Instead of fuzzy clouds or rigid bricks, imagine the object is made of smart, stretchy geometric shapes (like a 3D version of a polyhedral dice).
How it moves: These shapes have "vertices" (corners) that act like the joints of a puppet. When the object moves, the computer doesn't just shift the whole shape; it pulls and pushes these corners based on real physics.
The "Skinning" Trick: To make this efficient, the system uses a technique called Neural Skinning. Think of this like a digital puppeteer. Instead of calculating the physics for every single atom in the object (which takes forever), the system learns a few "master moves" (like bending, twisting, or squishing) and applies them to the whole shape. It's like how a video game character moves: you don't control every muscle; you control the bones, and the skin follows.

3. The Two-Step Magic

The system works in two main stages, like a two-act play:

Act 1: The "Snapshot" (Reconstruction)
First, the computer looks at the video from the very beginning (when the object is still) and builds a perfect 3D model of it. It figures out the shape, the color, and the texture. It's like taking a high-resolution photo and turning it into a 3D statue.

Act 2: The "Physics Class" (Simulation)
Next, the computer watches the rest of the video where the object is moving. It asks: "What physical rules made this object move like that?"

Is it hard like a rock?
Is it soft like a pillow?
Is it sticky like slime?

The system uses a Reduced-Order Simulation. Imagine trying to predict how a crowd of people will move. Instead of tracking 1,000 individuals, you track 10 "group leaders" whose movements represent the whole crowd. PhysConvex does this with physics: it uses a small set of "physics handles" to predict how the entire complex shape will deform, bounce, and collide.

4. Why It's a Big Deal

No More "Glitchy" Physics: Because the shapes are built on real math (continuum mechanics), they don't pass through walls or stretch infinitely. They behave like real matter.
Sharp Edges, Soft Jiggles: It can handle sharp corners (like a box) and soft squishes (like a balloon) in the same scene without blurring the edges.
Learning from Video: You don't need to tell the computer "this is rubber." It watches the video, sees how the object deforms, and figures out the material properties (like stiffness) all by itself.

The Bottom Line

Think of PhysConvex as a computer that doesn't just watch a video; it understands the physics inside it. It builds a 3D world out of smart, stretchy geometric shapes that know how to bounce, squish, and roll just like real objects, allowing us to simulate and reconstruct dynamic scenes with incredible accuracy and speed.

It's the difference between watching a cartoon of a bouncing ball and actually throwing a real rubber ball in a physics lab.

1. Problem Statement

Reconstructing and simulating dynamic 3D scenes that possess both visual realism and physical consistency remains a significant challenge in computer vision.

Limitations of Existing Methods:
- Purely Neural Approaches (e.g., NeRF, 3DGS): While excellent at appearance reconstruction, they often fail to capture complex material deformations and dynamics because they tightly couple geometric attributes with learned motion, disregarding underlying physical laws.
- Existing Physics-Informed Methods: Recent attempts to integrate differentiable simulators with neural primitives (like voxels in NeRF or ellipsoids in 3DGS) face critical limitations:
  - Lack of Spatial Sensitivity: Center-driven dynamics (updating primitives via central particles) struggle with non-uniform deformation.
  - Boundary Issues: Mesh-free approaches struggle to represent sharp or evolving boundaries, leading to inaccurate surface deformation.
  - Fixed Kernels: Predefined shapes (e.g., fixed ellipsoids) limit anisotropic modeling and produce poor spatial coverage near planar or angular regions.
Goal: To develop a unified framework that recovers appearance, geometry, and physical properties from multi-view videos while enabling mesh-free, physically consistent, and visually realistic simulation.

2. Methodology: PhysConvex

The authors propose PhysConvex, a framework that unifies visual rendering and physical simulation using 3D Dynamic Convex Radiance Fields governed by continuum mechanics.

A. Boundary-Driven Dynamic Convex Representation

Instead of center-driven particles, PhysConvex represents deformable radiance fields using 3D convex primitives (convex polytopes).

Explicit Vertex Dynamics: The deformation of a convex hull is modeled by advecting individual vertices under Newtonian motion. This provides spatial sensitivity to non-uniform deformations.
Implicit Surface Dynamics: The boundary is also represented implicitly via half-space support functions, allowing for explicit surface evolution and polyhedral structural refinement.
Advantages: This representation enables compact, gap-free volumetric coverage, handles sharp edges better than Gaussians, and supports spatially adaptive deformation.

B. Reduced-Order Convex Simulation

To efficiently simulate complex geometries and heterogeneous materials without explicit meshes or grids, the authors introduce a reduced-order simulation approach:

Neural Skinning Eigenmodes: The deformation is driven by spatially varying neural skinning eigenmodes ( $W_\theta$ ). These act as physics-informed, shape- and material-aware deformation bases.
Reduced Degrees of Freedom (DOFs): The simulation uses a low-dimensional vector of time-varying reduced DOFs ( $z(t)$ ) to control the deformation.
Physics Integration: The system solves for $z(t)$ using implicit time integration under Newtonian dynamics, minimizing potential energy (internal strain + external forces).
Differentiability: The entire pipeline (simulation and rendering) is differentiable, allowing physical parameters to be optimized end-to-end from video data.

C. Training Pipeline

The training occurs in two stages:

Static Reconstruction: Reconstruct the undeformed convex radiance field ( $C(0)$ ) from the first multi-view frame using image reconstruction losses ( $L_1$ , SSIM).
Dynamic Optimization: Jointly optimize elastic parameters (Young's modulus $E$ $E$ , Poisson's ratio $\nu$ $ν$ ) and the neural skinning field ( $W_\theta$ $W_{θ}$ ) using single-view video supervision. The system minimizes the difference between rendered and ground-truth pixel colors over time.
- Initialization: A feed-forward system (using a pretrained Video Vision Transformer) provides coarse initial predictions for physical parameters, which are then refined via differentiable simulation.

3. Key Contributions

PhysConvex Framework: A unified model integrating physical dynamics with deformable 3D convex radiance fields for video-based reconstruction and simulation.
Boundary-Driven Representation: A novel dynamic convex representation that combines vertex-level flexibility and surface-level evolution, enabling spatially adaptive and physically coherent deformation.
Reduced-Order Simulation: A mesh-free simulation method that advects dynamic convex fields using neural skinning eigenmodes as deformation bases, achieving high efficiency and accuracy.
State-of-the-Art Performance: Extensive experiments demonstrating superior accuracy in recovering geometry, appearance, and physical properties compared to existing methods (PAC-NeRF, Spring-GS, GIC, Vid2Sim).

4. Experimental Results

The method was evaluated on 12 high-quality 3D meshes (Google Scanned Objects) with complex geometries and textures.

Physical System Identification:
- PhysConvex achieved the lowest Mean Absolute Error (MAE) in estimating Young's modulus ( $E$ ) and Poisson's ratio ( $\nu$ ) across most test cases compared to baselines.
Dynamic Reconstruction:
- Outperformed baselines in PSNR (30.00 vs. ~22-28), SSIM (0.962 vs. ~0.92-0.95), and perceptual loss (FoVVDP).
- Visually, it preserved sharp details and dynamic fidelity, whereas other methods produced blurred textures or inaccurate dynamics.
Future State Prediction:
- Demonstrated superior temporal generalization, predicting future frames (8 frames ahead) with higher accuracy than competitors.
Generalization:
- Successfully generalized to novel materials (Elastic, Plasticine, Sand), new forces, and complex boundary conditions (e.g., sliding on moving floors).
Efficiency:
- Training Time: Average of 6 minutes per scene (vs. 32–69 minutes for baselines).
- Primitives: Requires fewer primitives (~~23k) compared to Gaussian-based methods (~~31k), thanks to the compact nature of convex coverage.

5. Significance and Impact

PhysConvex addresses the critical gap between visual rendering and physical simulation by moving away from fixed-shape primitives (like Gaussians) toward physically grounded convex primitives.

Mesh-Free & Grid-Free: It eliminates the need for explicit meshing or background grids, making it highly suitable for complex, evolving geometries.
Unified Optimization: It allows for the simultaneous recovery of appearance, geometry, and physical properties from video, a task previously difficult due to the decoupling of these attributes in other methods.
Practical Application: The method's efficiency and ability to handle diverse materials and boundary conditions make it a strong candidate for applications in robotics, virtual reality, and scientific simulation where both visual fidelity and physical accuracy are paramount.