IRIS: A Real-World Benchmark for Inverse Recovery and Identification of Physical Dynamic Systems from Monocular Video

1. Problem Statement

The paper addresses the challenge of unsupervised physical parameter estimation from monocular video. While recent methods have shown promise in estimating parameters for known governing equations, the field lacks a rigorous, high-fidelity real-world benchmark. Existing limitations include:

• Synthetic Bias: Most methods are evaluated on synthetic data with idealized lighting and geometry, failing to generalize to real-world complexities like motion blur and occlusion.
• Limited Scope: The only existing real-world dataset (Delfys75) is restricted to single-body dynamics, ignoring multi-body interactions (collisions, momentum transfer) which are central to classical mechanics.
• Lack of Protocol: There is no standardized protocol for evaluating governing equation identification, parameter accuracy, or extrapolation capabilities.
• Hidden Bugs: The authors identified a critical gradient-flow bug in the state-of-the-art latent-space pipeline (Garcia et al., Delfys75) that prevented physics parameters from receiving gradient updates.

2. Methodology: The IRIS Benchmark

The authors introduce IRIS (Interaction and Real-world Inverse physics Sequences), a comprehensive benchmark designed to overcome these gaps.

A. Dataset Specifications

• Scale & Quality: 220 real-world videos captured at 4K resolution (3840×2160) and 60 fps.
• Phenomena: Covers 8 dynamical systems, split into:
  • Single-body (5 types): Dropping ball, Falling ball, Sliding cone, Pendulum, Rotating cone.
  • Multi-body (3 novel types): Hitting cones (ball striking a pyramid), Two moving pendulums (simultaneous release and collision), One static/one moving pendulum (collision chain).
• Ground Truth: All videos include independently measured ground-truth parameters (e.g., length, mass, friction coefficients) with uncertainty estimates.
  • Direct Measurement: Geometric parameters (heights, angles, radii) measured with calipers/lasers.
  • Fitted Measurement: Damping and friction coefficients derived from trajectory fits with quantified uncertainty.
• Protocol: 10 repeated trials per setting, split 7/1/2 for train/val/test.

B. Evaluation Protocol

The paper defines a standardized, method-agnostic evaluation framework with five axes:

1. Parameter Accuracy: Mean Absolute Error (MAE) against ground truth.
2. Equation Selection: Accuracy in identifying the correct governing ODE family from a candidate bank.
3. Identifiability: Analysis of gradient norms and ODE residuals to determine if parameters can be uniquely learned.
4. Robustness: Consistency across different model configurations.
5. Extrapolation: Performance predicting dynamics beyond the training horizon.

C. Baselines & Strategies Evaluated

The authors evaluated several baselines to establish reference performance:

• Latent-Space Pipeline: Based on Garcia et al. [7], but with a critical correction to the Euler integrator. The original implementation omitted the acceleration term in the position update ($z_{t+1} = z_t + \Delta t \dot{z}_t$), severing the gradient flow to physical parameters. The corrected version includes the $\Delta t^2 \ddot{z}_t$ term.
• Multi-Step Loss: A variant enforcing agreement over $K$ future time steps to provide denser gradient signals.
• Equation Identification Strategies:
  1. VLM Temporal Reasoning: Using Vision-Language Models (GPT-4V, LLaVA, InternVL) to reason over sampled frames.
  2. Describe-then-Classify: A two-stage VLM pipeline (describe motion $\to$ classify equation).
  3. CNN Classifier: A ResNet-18 trained on the specific distribution (upper bound for in-distribution performance).
  4. Oracle: Path-based ground truth labels.

3. Key Contributions

1. IRIS Dataset: The first high-fidelity real-world benchmark combining multi-body interaction dynamics with ground-truth physical parameters and uncertainty estimates.
2. Standardized Protocol: A rigorous evaluation framework covering parameter accuracy, equation selection, identifiability, and extrapolation.
3. Diagnostic Findings:
   • Gradient-Flow Bug: Discovered and fixed a critical bug in the SOTA latent-space pipeline that rendered parameter learning impossible for certain dynamics (e.g., pendulum length estimation error dropped from 95m to 3.8m after the fix).
   • Multi-Step Instability: Demonstrated that while multi-step loss improves single-body dynamics, it causes catastrophic divergence on multi-body systems due to error accumulation in coupling terms.
4. Equation Identification Insights: Revealed that VLM strategy rankings reverse between datasets. "Temporal reasoning" leads on Delfys75, but "Describe-then-classify" leads on IRIS, suggesting that articulating complex dynamics in natural language aids disambiguation for multi-body scenarios.

4. Results & Analysis

Parameter Estimation

• Single-Body: The corrected baseline achieved near-perfect recovery for sliding cone angles (MAE = 0.00) and significant improvements for pendulum lengths.
• Multi-Body: The multi-step loss ($K=5$) failed catastrophically on multi-body scenarios, producing errors exceeding 100m for pendulum lengths (vs. ~1-3m for one-step). This highlights the difficulty of modeling contact forces with simplified, continuously active coupling coefficients in latent ODEs.
• Damping: Damping coefficients consistently showed higher relative error than conservative parameters, indicating they are harder to identify from short video clips.

Equation Identification

• CNNs: Achieved near-perfect accuracy (99.30% on IRIS) but require retraining for new phenomena.
• VLMs:
  • On Delfys75: Temporal reasoning (73.33%) > Describe-then-classify (61.11%).
  • On IRIS: Describe-then-classify (72.92%) > Temporal reasoning (65.00%).
  • Insight: The two-stage pipeline allows VLMs to articulate complex collision dynamics in natural language, which helps distinguish novel multi-body phenomena better than direct frame comparison.

Identifiability & Extrapolation

• Gradient Norms: The corrected integrator restored non-zero gradient norms, confirming parameter identifiability. Multi-body systems retained higher gradient norms at convergence, suggesting the simplified contact model does not fully explain the encoded trajectory.
• Extrapolation: Errors grew significantly with longer horizons and larger initial amplitudes (e.g., 90° pendulum), motivating the need for physics-constrained extrapolation losses.

5. Significance and Future Directions

The IRIS benchmark serves as a critical diagnostic tool for the field of physics-based vision.

• Realism Gap: It exposes the limitations of current latent-space methods when transitioning from synthetic/single-body to real-world/multi-body scenarios.
• Contact Modeling: The failure of multi-step losses on collisions suggests a need for contact-aware integration or truncated rollouts rather than simple horizon extension.
• VLM Utility: The results suggest that VLMs are promising for equation routing, particularly when leveraging their ability to describe complex physical interactions, though they currently lag behind supervised classifiers in accuracy.
• Open Challenges: The paper identifies open problems including the need for better contact models (moving beyond spring-like coupling), improved damping identifiability, and robust latent-to-SI calibration.

In summary, IRIS provides the necessary infrastructure to move beyond synthetic benchmarks, revealing systematic failure modes in current state-of-the-art methods and defining a clear path for future research in inverse physics recovery.