Technical Summary: BilliardPhys-Bench

Problem Statement

Current Multimodal Large Language Models (MLLMs) demonstrate proficiency in static image recognition and semantic understanding but struggle with intuitive physical reasoning. Specifically, predicting the future evolution of objects—such as trajectories, collisions, and final resting positions—from a single static image remains a significant challenge. Existing benchmarks often focus on declarative knowledge of physical laws (Static QA), abstract symbolic reasoning, or interactive trial-and-error agents. There is a notable gap in evaluating single-frame forward simulation, where a model must infer dynamic physical outcomes from a static initial condition without iterative interaction. This gap highlights a separation between perception (describing a scene) and reasoning (predicting how that scene evolves).

Methodology

Benchmark Design: BilliardPhys-Bench

The authors introduce BilliardPhys-Bench, a benchmark designed to test physical reasoning in synthetic billiard environments. The benchmark utilizes a procedural engine to generate randomized scenarios featuring:

• Physics Model: A constant friction model ($\mu = 0.002$) and perfectly elastic collisions (ball-to-ball and ball-to-wall).
• Data Generation: A three-stage pipeline:
  1. Initial-condition sampling: Randomizing cue ball velocity and object ball positions.
  2. Physical simulation: A high-fidelity engine computes motion, friction-based deceleration ($v(s) = \sqrt{v_0^2 - 2\mu gs}$), and collisions, exporting ground truth annotations.
  3. Visualization: Rendering high-resolution initial-state images with velocity vectors and coordinate axes.

Diagnostic Tasks

The benchmark decomposes physical reasoning into three distinct tasks, evaluated across temporal horizons of 1 to 5 seconds:

1. Task 1 (Discrete Collision Prediction): Determining if the cue ball will collide with specific object balls (True/False).
2. Task 2 (Wall Interaction Reasoning): Predicting which table walls the cue ball will collide with (Multi-select).
3. Task 3 (Precise Coordinate Estimation): Predicting the exact 2D coordinates ($x, y$) of all balls at the target time, accounting for pocketed balls (null).

Evaluation Pipeline

The study evaluates leading proprietary and open-source MLLMs (from GPT, Claude, Gemini, and Qwen families). Models are prompted with the initial scene image and a structured prompt detailing physical constants (friction, gravity, elasticity) and task definitions.

• Output Format: Models are required to return structured JSON outputs.
• Reasoning Strategy: Chain-of-thought (CoT) is enabled for Tasks 1 and 2 but disabled for Task 3 to prevent interference with precise coordinate formatting.
• Scoring: Correctness is judged against ground truth: exact match for wall collisions, per-ball collision labels for Task 1, and Euclidean distance $\le$ ball radius for Task 3. A weighted total score ($S_{total}$) combines the three tasks, weighting Task 3 highest (0.4) due to its precision requirements.

Key Contributions

1. Procedurally Generated Multi-Tier Benchmark: A dataset of randomized billiard scenarios paired with physics-engine ground truth, specifically designed to test discrete event prediction, continuous state estimation, and complex interaction chains.
2. Diagnostic Framework for Failure Modes: The study identifies and analyzes specific failure patterns in MLLMs, most notably "stasis bias" (the tendency to predict no interaction when the correct physical outcome is complex) and the decoupling of event reasoning from terminal-state prediction.
3. Empirical Analysis of Inductive Biases: The results highlight the limitations of next-token prediction for physical simulation and suggest the need for hybrid neural-physical approaches, factor graphs, or differentiable physics engines to improve physical inductive biases.

Results

Performance Trends

• Temporal Degradation: Performance across all models declines as the simulation horizon increases from 1s to 5s. Longer horizons introduce more complex interaction chains and cumulative friction effects, causing small initial errors to compound.
• Model Tiering:
  • Top Tier: GPT-5.5 achieved the highest overall score (73.58%), followed closely by GPT-5.4-Pro (72.29%). These models demonstrated balanced performance across all three tasks.
  • Specialized Performers: Qwen3.6-Plus showed strong event reasoning (Task 1 and 2) but weaker terminal-state prediction (Task 3). Conversely, GPT-5.4 and Claude-Opus-4.7 exhibited strong spatial grounding (Task 3) but failed significantly on discrete event identification (Task 1 and 2).
  • Stasis Bias: Models with weaker event reasoning often defaulted to conservative predictions (predicting no collisions) as uncertainty increased, even when their final position estimates remained plausible.

Specific Findings

• Event vs. State: There is a clear dissociation between a model's ability to identify events (collisions) and its ability to estimate states (final positions). Strong terminal-state prediction does not guarantee accurate event reasoning.
• Compute and Reasoning: "Pro" variants of models (e.g., GPT-5.4-Pro) generally outperformed their base counterparts, particularly in event inference, likely due to deeper test-time reasoning (consuming ~8x more tokens).
• Family Differences: The Claude family showed non-monotonic progression, with later variants (4.7) inverting the strengths of earlier ones (4.5/4.6), shifting from strong wall-interaction reasoning to strong terminal-state prediction but weak event detection.

Significance and Claims

The paper claims that BilliardPhys-Bench provides a direct mechanism to evaluate the limitations of current MLLMs in visual dynamics. The findings suggest that:

1. Current Limitations: MLLMs lack a stable internal model of physical dynamics over time, particularly when dealing with sequential collisions and friction.
2. Architectural Needs: The observed "stasis bias" and the gap between event reasoning and state prediction indicate that standard next-token prediction architectures are insufficient for robust physical simulation.
3. Future Directions: The benchmark points toward the necessity of integrating stronger physical inductive biases, potentially through hybrid neural-physical approaches, to bridge the gap between scene recognition and predictive reasoning.

The authors maintain a modest scope, acknowledging that the benchmark uses simplified 2D physics (constant friction, perfect elasticity) and does not generalize to complex real-world dynamics like spin or deformable objects. The benchmark is intended as a diagnostic tool to measure progress toward models capable of reliable physical reasoning.