Technical Summary: PhysAgent

Problem Statement

Achieving fully automated, physically plausible 4D motion synthesis remains a critical bottleneck in computer graphics and generative AI. While recent methods have advanced the optimization of individual object materials, they fail to address the dynamic evolution of diverse scenes governed by environmental force fields. Currently, configuring these complex force fields relies entirely on manual expert intervention, hindering large-scale simulation data generation.

Existing automated approaches face two primary limitations:

1. Traditional Optimization (SDS): Methods relying on differentiable simulators and Score Distillation Sampling (SDS) suffer from sluggish gradient backpropagation, are prone to local optima, and are mathematically incapable of dynamically switching discrete variables (e.g., toggling between different force field types).
2. Naive LLM Prompting: Directly mapping text to physical parameters using Large Language Models (LLMs) lacks underlying simulation feedback. Without grounding, these models fail to comprehend the complex numerical characteristics of the Material Point Method (MPM), leading to severe physical distortions and "hallucinations."

Methodology: PhysAgent Framework

PhysAgent introduces a "simulator-in-the-loop" multi-agent framework that accepts joint text-and-image multimodal inputs to synthesize 4D physical dynamics. The system decouples intrinsic material properties (derived from images) from extrinsic environmental dynamics (driven by text), optimizing the latter through a closed-loop reasoning process.

1. Semantic Agent & Initialization

• Input Processing: The system extracts intrinsic materials and spatial anchors from a reference image using PhysGM. Simultaneously, a Semantic Agent interprets the user's text prompt.
• Force Field Skill Module: To prevent physical hallucinations, the Semantic Agent is equipped with an externalized Force Field Skill Library. This module explicitly defines the taxonomy, mathematical formulations, and operational constraints of supported forces.
• Output: The agent generates a structured Force Field JSON which is deterministically fused with the 3D Gaussian Splatting (3DGS) representation and spatial anchors to create an initial, physically compliant simulation configuration ($\Theta_0$).

2. Forward Physical Simulation

The framework utilizes the Material Point Method (MPM) as the forward execution engine.

• Simulation Loop: MPM discretizes the continuum body into Lagrangian particles, performing Particle-to-Grid (P2G) transfers, solving Newton's second law on the grid with injected external forces ($f_{ext}$), and updating particle states via Grid-to-Particle (G2P) transfers.
• Rendering: The simulator drives the 3DGS to render a 2D physical video sequence ($V$).

3. Trajectory-Grounded Multi-Agent Feedback

This is the core optimization mechanism, replacing traditional gradient descent with a reasoning-driven loop:

• Visual Perception: The system treats the simulator as a black-box 2D renderer. It employs vision foundation models (SAM 2 for masks, Depth Anything V3 for depth, and CoTracker3 for tracking) to extract dense 3D motion trajectories ($\mathcal{T}$) from the rendered video.
• Structured Feedback: These trajectories are synthesized into Structured Motion Information ($F_k$), serving as explicit visual priors orthogonal to semantic reasoning.
• Refine Agents:
  • Planning Agent: Analyzes the deviation between the extracted motion $F_k$ and the original text intent $P_{text}$ using physical commonsense reasoning.
  • Execution Agent: If deviations exist, the agent updates the force field configuration ($\Theta$). Unlike SDS, which relies on local gradient updates, the Execution Agent leverages global LLM reasoning to execute zero-shot macroscopic leaps. This allows the system to escape local optima and dynamically switch discrete force field types without sluggish backpropagation.

Key Contributions

1. First Simulator-in-the-Loop Multi-Agent Framework: PhysAgent is the first framework to automate diverse 4D physical synthesis by explicitly decoupling image-derived intrinsic priors from text-driven extrinsic dynamics, enabling rapid generation of complex force field configurations.
2. Force Field Skill Module: The introduction of an externalized skill library enables the Semantic Agent to generate physically plausible initializations, resolving the inability of standard LLMs to comprehend MPM numerical characteristics.
3. Trajectory-Grounded Refinement: By utilizing vision foundation models to extract explicit motion trajectories and feeding them back to LLM agents, the system bypasses the sluggish optimization of SDS. This mechanism enables zero-shot macroscopic state leaps to escape local optima and handle discrete force switching.
4. Performance Superiority: Extensive experiments demonstrate that PhysAgent significantly outperforms existing baselines (including SDS-based methods and naive LLM approaches) in both generation diversity and physical accuracy.

Experimental Results

The authors evaluated PhysAgent on the PhysGaussian and Objaverse datasets, comparing against state-of-the-art methods like PhysSplat, DreamPhysics, and OmniPhysgs.

• Quantitative Metrics: PhysAgent achieved the highest scores in CLIPsim (semantic alignment) and User Preference Rating (UPR) across various force types (drop, stretching, sway). Notably, it showed substantial gains in complex interactive tasks where baselines struggled with discrete force attributes.
• Qualitative Analysis: Visual comparisons revealed that baseline methods often produced physical distortions or erroneous force applications under complex instructions. In contrast, PhysAgent generated smooth, physically plausible deformations.
• Ablation Studies:
  • Removing the Force Field Skill Library caused a precipitous drop in performance, confirming the necessity of explicit physical rule constraints to prevent parameter hallucinations.
  • Removing the Refine Agents (reverting to open-loop generation) resulted in suboptimal dynamics and kinematic drifting, validating that trajectory-grounded feedback is pivotal for high-fidelity synthesis.

Significance and Claims

The paper claims that PhysAgent represents a significant step toward fully automated, physically grounded 4D synthesis. By bridging the modality gap between high-level user intent and low-level physical parameters, the framework eliminates the need for manual expert intervention in configuring environmental force fields.

The authors assert that their approach overcomes the mathematical rigidity of backpropagation-based optimization and the physical inaccuracies of ungrounded LLMs. By formalizing a "simulator-in-the-loop" paradigm, PhysAgent enables the generation of stable, diverse, and physically accurate dynamic scenes from arbitrary multimodal prompts, establishing a new frontier for automated simulation data generation in embodied AI and computer graphics.