PhysForge: Generating Physics-Grounded 3D Assets for Interactive Virtual World

Technical Summary: PhysForge

Problem Statement
The rapid advancement of 3D generative models has enabled the synthesis of high-fidelity geometric assets, yet a critical bottleneck remains for embodied AI and interactive virtual worlds: the lack of physics-grounded generation. Existing methods predominantly produce "hollow shell" assets—static geometries and textures that lack the functional properties, material definitions, and kinematic parameters necessary for physical interaction. Consequently, these assets cannot be grasped, pushed, or manipulated by agents in simulation environments. Current approaches to part-aware generation often focus on visual decomposition rather than functional logic, while physics-based generation methods frequently rely on predefined content repositories or struggle to generalize to novel object categories with accurate articulation.

Methodology: PhysForge Framework
To address these limitations, the authors propose PhysForge, a decoupled two-stage framework designed to generate 3D assets rooted in functional logic and hierarchical physics. The system is supported by PhysDB, a newly constructed large-scale dataset.

1. PhysDB (The Data Foundation):
   The authors introduce PhysDB, a dataset of 150,000 assets sourced from Objaverse, PartNet-Mobility, and Infinite-Mobility. It features a novel four-tier annotation system:

   • Holistic Tier: Global properties (real-world scale, usage scene).
   • Static Properties Tier: Part-level attributes (semantic labels, materials, mass).
   • Functional Tier: Intrinsic functions (e.g., "to contain") and state machines (e.g., [open, closed]).
   • Interactive Tier: Kinematic definitions including joint types (revolute, prismatic, fixed), parent-child relationships, and precise parameters (axis origin, direction, limits).

2. Stage 1: VLM-Based Planning (The "Physical Architect"):
   The first stage utilizes a Vision-Language Model (VLM), specifically finetuned Qwen2.5-VL, to act as a planner. Instead of generating geometry directly, the VLM takes a single input image, optional 2D masks, and 3D voxel features as input to generate a "Hierarchical Physical Blueprint."

   • This blueprint defines the bounding box layout for all parts and their detailed physical properties.
   • The model is trained to co-predict physical constraints (material, function) alongside spatial bounds. The authors observe a synergistic effect: providing physical constraints significantly aids the model in resolving part granularity ambiguity, allowing for coherent part decomposition even without 2D mask guidance.
   • The output includes hierarchical part structures, parent nodes, articulation types, and quantized bounding box coordinates.

3. Stage 2: Diffusion-Based Generation (The "Realization"):
   The second stage employs a diffusion model to synthesize high-fidelity geometry, textures, and precise kinematic parameters based on the blueprint.

   • KineVoxel Injection (KVI): A core innovation where articulation parameters (origin, axis, limits) are encoded into a special "KineVoxel" representation.
   • This KineVoxel is injected into the diffusion pipeline alongside standard geometry latents. The system uses independent Kinematic Encoders and Decoders to map kinematic data into a unified latent space shared with geometry latents within a middle transformer.
   • A Joint Type Embedding (derived from the VLM's plan) is added to the KineVoxel to guide the diffusion process, ensuring the generated geometry aligns with the planned functional constraints (e.g., ensuring a "revolute" joint allows rotation).
   • The model is trained using Conditional Flow Matching (CFM) with a composite loss function that weights kinematic parameter accuracy heavily ($\lambda_{kine} = 10$).

Key Contributions

• Formulation and Framework: The proposal of a novel "VLM Planning + Diffusion Generation" paradigm that decouples high-level physical planning from low-level geometric realization.
• PhysDB Dataset: The creation of a large-scale (150k assets), part-aware dataset with fine-grained, four-tier physical annotations, filling a critical data gap in the field.
• KineVoxel Injection: A mechanism enabling the synergistic generation of geometry and precise kinematic parameters within a unified diffusion framework.
• Validation: Extensive experiments demonstrating state-of-the-art performance in both part structure planning and physics-grounded generation.

Experimental Results
The paper evaluates PhysForge on several benchmarks, including PhysXNet, PartObjaverse-Tiny, and custom splits from PhysDB.

• Part Structure Planning: On the PartObjaverse-Tiny dataset, PhysForge achieves superior BBox IoU (42.95%) and Voxel IoU (53.74%) compared to baselines like OmniPart and PartField. Notably, the model performs robustly even without 2D mask inputs, outperforming baselines that rely on SAM-generated masks, validating the efficacy of physics-guided planning.
• Physics Property Generation: On PhysXNet and PhysDB, PhysForge outperforms baselines (TRELLIS, PhysXGen) in geometric quality (lower Chamfer Distance, higher F1-Score) and the accuracy of predicted physical properties (material, affordance, function).
• Kinematic Parameter Accuracy: In articulated object generation, PhysForge significantly reduces Joint Axis Error (0.101 vs. 0.241 for Singapo) and Joint Pivot Error (0.071 vs. 0.153) compared to state-of-the-art methods like Articulate Anything and URDFormer.
• Ablation Studies: Removing the joint type embedding or the dedicated kinematic encoder leads to a marked degradation in joint accuracy, confirming the necessity of the two-stage synergy and the KVI mechanism.

Significance and Applications
The authors claim PhysForge provides a robust data engine for interactive 3D content. The generated assets are "simulation-ready," meaning they can be directly deployed in:

• Robotic Simulators: Enabling realistic manipulation tasks where agents interact with functional parts (e.g., RoboTwin).
• Interactive Virtual Worlds: Allowing for complex, physics-based interactions in game engines (e.g., Unity, Unreal Engine) without manual rigging.
• Agent-Environment Interaction: Facilitating natural language queries where embodied agents can retrieve physical blueprints to plan manipulation tasks.

The paper concludes that by shifting focus from holistic shape generation to physics-centric synthesis, PhysForge bridges the gap between static 3D assets and the dynamic requirements of embodied AI.