Technical Summary: DiscoverPhysics

Problem Statement

While frontier Large Language Models (LLMs) demonstrate strong performance on static benchmarks of scientific knowledge (e.g., GPQA) and reasoning tasks, their ability to perform genuine scientific discovery remains unproven. Existing evaluations often rely on memorized facts, symbolic regression on fixed datasets, or interactive tasks where the underlying physical laws are canonical with only shifted parameters. These approaches fail to test the core of scientific discovery: the ability to generate novel hypotheses, design informative experiments to distinguish between competing theories, and revise conceptual models based on noisy observations. Specifically, current benchmarks do not adequately probe whether agents can uncover latent structures (such as hidden particle species or extra dimensions) or distinguish signal from noise in non-canonical physical environments.

Methodology

The authors introduce DISCOVERPHYSICS, an interactive benchmark designed to evaluate LLM agents on the discovery of physical laws in simulated worlds that deliberately deviate from standard textbook physics.

Benchmark Structure

• Simulated Worlds: The benchmark consists of 22 distinct worlds (11 public, 11 private) governed by non-canonical force laws. Examples include screened (Yukawa) and fractional-power gravity, multi-species couplings, hidden dark-matter-like particles, non-coordinate-free physics, time-varying interactions, and extra dimensions.
• Agent Interaction: An LLM agent is tasked with discovering the hidden law of a specific world. The agent operates within a fixed budget of experimentation rounds ($n$).
• The Loop:
  1. Experiment Proposal: The agent proposes test particles defined by initial positions, velocities, charges, and measurement times.
  2. Simulation: An $N$-body simulator (using integrators like Yoshida4 or RK4) generates raw trajectory data based on the hidden ground-truth law.
  3. Hypothesis Refinement: The agent analyzes the returned trajectory data, proposes candidate laws, and optionally uses a tool to fit parameters via scipy.optimize.
  4. Iteration: This cycle repeats until the budget is exhausted.
• Final Output: The agent must submit two items:
  1. A natural-language explanation of the world's physics.
  2. A Python implementation of the inferred law (a function to evolve particles).

Evaluation Metrics

The benchmark employs a dual-axis evaluation framework to assess both predictive accuracy and conceptual understanding:

1. Trajectory MSE: The mean squared Euclidean distance between the agent's predicted trajectories and the ground-truth trajectories on held-out test particles (noise-free). This measures predictive accuracy.
2. Explanation Score: An LLM judge (Claude Opus 4) evaluates the agent's natural-language explanation against a human-written rubric (scored 0–10). This measures conceptual understanding and the ability to identify structural features (e.g., screening, hidden species).
3. Pass Criteria: A world is considered "passed" if the normalized trajectory MSE is below 0.1 (10% error) and the explanation score is $\ge 0.9$.

Experimental Setup

The authors evaluated 11 frontier and open-source LLMs (including proprietary models like Claude Opus 4.7, GPT-5.5, and open models like Qwen3.5 and Llama-3.3). Experiments were run across 5 random seeds per model per world. The study also included ablation studies comparing agent-guided experimentation against randomized experimentation and varying levels of observational noise.

Key Results

• Overall Performance: Even the strongest frontier agents (Claude Opus 4.7 and GPT-5.5) pass only approximately 50% of the worlds within five attempts. Performance drops significantly on worlds requiring the identification of latent structure (e.g., hidden dark matter, extra dimensions).
• Predictive Accuracy vs. Conceptual Understanding: The metrics decouple. GPT-5.5 achieves the lowest trajectory MSE (highest predictive accuracy) but often scores lower on explanation quality compared to Claude Opus 4.7. This suggests GPT-5.5 tends to fit data well by adopting flexible equations early but fails to revise its conceptual picture, whereas Claude Opus 4.7 demonstrates better hypothesis refinement through experimentation.
• Experimental Design: There is a significant gap between guided and randomized experimentation for top models. Claude Opus 4.7 shows monotonic improvement in pass rate and explanation scores as the number of experimental rounds increases, indicating it learns from new data and designs informative experiments. In contrast, open-source models show little difference between guided and random modes, suggesting they do not effectively utilize their experimental budget.
• Open vs. Proprietary: Open-source models lag substantially behind commercial models in both designing experiments and extracting conclusions. The best open model (Qwen3.5) solves only ~10% of worlds in five attempts.
• Failure Modes: Analysis of failure modes reveals distinct weaknesses:
  • GPT-5.5: Failures concentrate on "considering the right kind of law" (missing latent structure) and handling numerical singularities in ODE integrators.
  • Claude Opus 4.7: Failures are more distributed across experimental design (failing to discriminate between hypotheses), writing code that faithfully implements verbal reasoning, distinguishing signal from noise, and knowing when to stop experimenting.

Significance and Claims

The paper claims that DISCOVERPHYSICS represents the strongest current test of LLMs' ability to perform the ill-defined, iterative research process of hypothesis generation and refinement based on self-proposed experiments.

• Beyond Memorization: The benchmark demonstrates that high performance on knowledge-based benchmarks does not guarantee the ability to discover novel physical laws.
• The Necessity of Experimentation: The results highlight that scientific discovery in physical sciences relies critically on the ability to design insightful experiments, not just on reasoning over static data.
• Dual Evaluation Necessity: The paper argues that predictive accuracy (MSE) alone is insufficient for evaluating scientific discovery; a model can fit data perfectly without understanding the underlying principles. The dual evaluation framework captures whether agents merely interpolate data or identify the generative mechanisms.
• Current Limitations: The authors modestly conclude that while frontier models represent a substantial improvement, they are not yet capable of fully autonomous scientific discovery, particularly when faced with non-canonical physics requiring the inference of hidden degrees of freedom. The benchmark serves as a tool to measure progress in this specific, high-difficulty domain.