🤖 AI

ExPhy: A Benchmark for Explicit Physical Property Learning in Multi-Object Trajectory Forecasting

This paper introduces ExPhy, a new benchmark for multi-object trajectory forecasting that explicitly evaluates physical property learning (mass, friction, and restitution) alongside trajectory prediction, demonstrating through the proposed PhyODE model that incorporating explicit physical property estimation significantly improves long-horizon out-of-distribution forecasting performance.

Original authors: Rui Wang, Yeteng Wu, Xianlin Zhang, Mengshi Qi

Published 2026-08-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rui Wang, Yeteng Wu, Xianlin Zhang, Mengshi Qi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

To understand how the world moves, humans rely on an intuitive sense of physics. We know that a heavy stone will roll differently than a light feather, or that a rubber ball will bounce higher than a lump of clay, even without calculating the forces involved. This ability to predict future motion based on the hidden properties of objects is a fundamental part of how we interact with our environment. For decades, scientists have tried to teach computers this same skill, building systems that can watch a scene and guess where objects will go next. However, a significant gap has existed in how these systems are tested. Most existing tests only ask the computer to draw a line showing where an object will be in the future. If the line is close to the truth, the computer gets a good score, even if it arrived at that answer by guessing or by using a trick that has nothing to do with real physics. The computer might predict the right path without ever understanding that the object is heavy, or that the floor is slippery. This leaves researchers unsure if the machine is truly reasoning about the physical world or just memorizing patterns.

A team of researchers has addressed this blind spot by creating a new testing ground called ExPhy. This benchmark is a collection of 24,000 simulated scenes where objects collide, slide, and bounce. Unlike previous tests, ExPhy does not just hide the future path of the objects; it also provides the specific, hidden numbers that govern their behavior. For every object in the simulation, the system knows its exact mass, how much friction it has against the ground, and how bouncy it is. The researchers used this setup to see if a computer model could not only predict the future path of the objects but also correctly identify these hidden physical properties just by watching them move. They found that while a model can learn to draw a very accurate path, it does not necessarily mean it has learned the correct physical rules. A model can be right about where an object goes but completely wrong about why it goes there.

To test this, the researchers built a specific computer model named PhyODE. This model was designed to work in two steps. First, it watches the observed movement of the objects and tries to guess their mass, friction, and bounciness. It does not see these numbers directly; it has to infer them from the way the objects accelerate, slow down, or collide. Once the model makes these guesses, it uses them to calculate the future motion. This approach is different from other models that simply look at the past path and try to draw the next part of the line without ever naming the physical properties. The researchers trained PhyODE on a large set of scenes and then tested it on new scenes where the objects had different weights or were placed in different starting positions. They wanted to see if the model could handle these changes, a situation known as out-of-distribution testing, where the rules of the game have shifted slightly from what the computer saw during training.

The results revealed a surprising distinction between predicting a path and understanding physics. When the researchers looked at how well the models predicted the future trajectories, PhyODE performed very well, often beating other advanced systems. In tests where the objects started in new, unseen positions, PhyODE reduced the error in its long-term predictions by about 31 percent compared to the strongest competing models. However, when the researchers checked if the model had correctly identified the mass or friction of the objects, the picture was more complex. They found that a model could achieve a very low error in predicting the path while still having a high error in estimating the physical properties. This proves that getting the destination right does not guarantee that the computer understands the journey. The study suggests that accurate trajectory forecasting and accurate physical property estimation are related but separate skills. A system can be a good navigator without being a good physicist.

The researchers also tested how well their model could transfer its knowledge to a completely different set of data, a benchmark called ComPhy, which focuses on hidden mass and charge in video reasoning. Without any extra training on this new data, PhyODE performed better than all other models tested. This suggests that the way the model was built to explicitly estimate physical properties helped it learn a more robust understanding of motion that could apply to different scenarios. The study concludes that to truly understand object dynamics, we must evaluate both the path the object takes and the physical reasons behind it. By providing a benchmark that demands both, ExPhy offers a clearer way to measure whether artificial intelligence is truly learning the laws of physics or just learning to mimic the results. The work highlights that in the quest to teach machines about the physical world, getting the right answer is not enough; we must also ensure they are using the right reasoning to get there.

Technical Summary: ExPhy and PhyODE

Problem Definition
Understanding object dynamics requires predicting future trajectories while accounting for the physical properties (mass, friction, restitution) that govern motion and interaction. While existing benchmarks in physical reasoning focus heavily on future-outcome prediction, event plausibility, or causal reasoning, they rarely expose object-level physical properties as explicit, continuous-valued evaluation targets alongside trajectory forecasting. Consequently, a model may achieve low trajectory error (e.g., low ADE/FDE) without accurately recovering the underlying physical parameters, creating a gap between predictive accuracy and physical understanding. This paper addresses the need to jointly evaluate trajectory forecasting and explicit object-level property estimation.

Methodology
The authors introduce ExPhy, a benchmark, and PhyODE, a physics-guided hybrid model designed to operate within this framework.

ExPhy Benchmark:
- Dataset: Comprises 24,000 simulated dynamic scenes generated using the PyBullet physics engine. Scenes contain 3–8 interacting rigid bodies (cubes, cylinders, spheres) with diverse geometric shapes.
- Annotations: Unlike prior works, ExPhy provides explicit, continuous-valued ground-truth labels for three physical properties per object: mass ( $m$ ), lateral friction coefficient ( $\mu$ ), and restitution ( $e$ ).
- Evaluation Protocols: The benchmark defines three splits to test generalization:
  1. In-Distribution (ID): ExPhy-A, with standard interpolation.
  2. OOD-Parameter: ExPhy-B, where physical property distributions (mass, friction, restitution) are shifted outside the training ranges.
  3. OOD-Initial: ExPhy-C, where initial states (locations and velocities) are shifted while property distributions remain consistent.
- Task: Models observe trajectories ( $X_{obs}$ ) and must predict future trajectories ( $\hat{X}_{pred}$ ) and estimate physical properties ( $\hat{p}$ ), though property labels are never provided as inference inputs.
PhyODE Model:
- Architecture: A hybrid model coupling an explicit property estimator ( $F_\psi$ ) with a differentiable trajectory solver ( $G_\phi$ ).
- Property Estimator ( $F_\psi$ ): Extracts cues from observed trajectories via three branches:
  1. Raw Motion: Encodes coordinate-level temporal evolution (positions, velocities, accelerations).
  2. Invariant Descriptor: Summarizes trajectory geometry independent of absolute coordinates.
  3. Pairwise Relations: Captures interaction-dependent cues using attention over relative positions and velocities.
    These features are fused and decoded to predict mass, friction, and restitution.
- Trajectory Solver ( $G_\phi$ ): A physics-based hybrid dynamics solver that rolls out future states in continuous time. It combines:
  1. Physics Branch ( $f_{DPE}$ ): A property-conditioned module handling kinetic friction, gravitational acceleration, and impulse-based collision updates (using estimated mass, friction, and restitution).
  2. Residual Branch ( $f^{res}_\phi$ ): A learnable Neural ODE component that corrects translational and angular derivatives to account for unmodeled dynamics.
- Training: The model is trained end-to-end using a joint objective: a trajectory loss ( $L_{traj}$ ) and a property loss ( $L_{prop}$ ), normalized by property-specific scales.

Key Results

Trajectory Forecasting: PhyODE achieves state-of-the-art performance across all splits. Notably, on the challenging OOD-Initial (ExPhy-C) setting with long horizons (30–60 steps), PhyODE reduces Average Displacement Error (ADE) and Final Displacement Error (FDE) by 33.1% and 31.0%, respectively, compared to the strongest baseline.
Cross-Benchmark Transfer: In zero-shot evaluations on ComPhy (an independent benchmark for hidden mass and charge), PhyODE outperforms all baselines across short, mid, and long horizons, demonstrating that dynamics learned on ExPhy generalize to unseen scene distributions.
Property Estimation: Dedicated property predictors (e.g., Object-GNN) achieve the lowest property estimation error. Crucially, PhyODE variants trained only on trajectory loss show significantly degraded property estimation, while joint training improves both but does not match the accuracy of direct property supervision.
Ablation: Removing the explicit physics component causes the most significant performance drop, particularly in OOD-Initial settings, confirming the necessity of property-conditioned dynamics for extrapolation.

Significance and Claims
The paper claims that accurate trajectory forecasting does not necessarily imply accurate recovery of underlying physical properties. By introducing ExPhy, the authors provide a benchmark to jointly evaluate these two dimensions with explicit, continuous property labels, addressing the gap where existing benchmarks rarely expose such properties as dedicated evaluation targets. The results demonstrate that while trajectory error and property accuracy are related, they are distinct evaluation dimensions. The proposed PhyODE model validates that explicitly estimating physical properties and integrating them into a differentiable solver improves generalization, particularly for long-horizon forecasting under distribution shifts in initial states. The work underscores the value of explicit physical property learning for robust multi-object dynamics prediction.

Technical Summary: ExPhy and PhyODE

More like this