MaD Physics: Evaluating information seeking under constraints in physical environments

Technical Summary: MaD Physics – Evaluating Information Seeking Under Constraints in Physical Environments

Problem Statement
Scientific discovery is inherently a resource-constrained process, requiring agents to navigate trade-offs between the quality and quantity of measurements due to physical, temporal, and financial limitations. While existing benchmarks for scientific discovery focus on static knowledge-based reasoning (e.g., question-answering based on memorized facts) or unconstrained experimental design, they fail to evaluate an agent's ability to make strategic measurements and plan under strict constraints. Furthermore, standard benchmarks often rely on well-known physical laws, allowing agents to succeed through knowledge retrieval rather than empirical discovery. There is a critical gap in evaluating agents that must actively gather information, manage a finite budget, and infer underlying dynamics from noisy, sparse data in environments where the governing laws may be unknown or altered.

Methodology: The MaD Physics Benchmark
To address this gap, the authors propose MaD Physics (Measuring and Discovering Physics), a benchmark designed to evaluate the ability of agents to make informative measurements and draw conclusions subject to constraints on measurement quality and quantity.

• Environment Design: The benchmark consists of three distinct parametric environments based on Classical Mechanics, Fluid Mechanics, and Quantum Mechanics. To prevent agents from relying on pre-trained knowledge of standard physics, each environment includes "altered" physical laws (e.g., anisotropic inertial mass, modified gravity laws like $1/R$ or ripple effects, non-linear entanglement initialization, and generalized Born rules with $L_p$-norms).
• Evaluation Protocol: The evaluation occurs in two phases:
  1. Measurement Phase: The agent operates under a fixed budget $B$. It must sequentially choose what to measure (e.g., specific particles or coordinates), when to measure (time step), and with what fidelity (noise level). Each observation incurs a cost $C(o_k, \sigma_k)$, which increases with fidelity (lower noise). The agent must maximize information gain within the budget.
  2. Prediction Phase: Once the budget is exhausted or a maximum time $T_{max}$ is reached, the agent must infer the underlying physical law and predict the system's state at a future query time $T_{query} > T_{max}$.
• Agent Implementation: The authors evaluate four Gemini models (2.5 Flash Lite, 2.5 Flash, 2.5 Pro, and 3 Flash) using a minimal agent scaffold with code execution capabilities. Two prompting strategies are tested: a "Base" prompt providing general context and a "Strategy" prompt inspired by Bayesian experimental design, instructing the agent to maximize Expected Information Gain (EIG) per unit cost.

Key Contributions

1. Novel Benchmarking Methodology: The paper introduces a framework for evaluating information-seeking agents in physical domains that explicitly emphasizes the cost-fidelity trade-off in experimental design, moving beyond static reasoning tasks.
2. Parametric Environments with Altered Laws: The suite of three environments (Classical, Fluid, Quantum) includes specific, documented alterations to standard physical laws. This ensures agents must engage in active discovery rather than retrieving memorized knowledge.
3. Comprehensive Benchmarking of Frontier Models: The authors benchmark multiple Gemini models on MaD Physics, analyzing performance across different degrees of law alteration and demonstrating the impact of strategic prompting on navigating scientific uncertainty.

Empirical Results

• Model Performance: Generally, performance improves with model capability (from Flash Lite to Pro to 3 Flash). However, even the most capable models struggle with altered configurations.
• Prediction Accuracy: Agents often produce predictions that are out of bounds or inaccurate, even in normal physics configurations. While the "Strategy" prompt generally improves performance by enforcing systematic data collection, significant errors persist.
• Symbolic vs. Predictive Accuracy: A key finding is the disconnect between predictive accuracy and symbolic recovery. Agents frequently fail to identify the correct symbolic form of the underlying physical law (e.g., failing to distinguish between $1/r^2$ and $1/r$ gravity), even when they can make reasonable predictions.
• Multimodality and In-Context Learning:
  • Visual Observations: When agents are provided with visual renderings instead of numerical data, prediction errors increase significantly, though trends regarding model capability remain consistent.
  • In-Context Learning: Over multiple trials, Gemini 3 Flash demonstrates the ability to reduce prediction error, whereas Gemini 2.5 Pro fails to learn in altered configurations.
• Parameter Inference: In a variant where the structural form is known and the goal is to infer a single parameter ($\kappa$), agents tend to underestimate the parameter, showing a bias toward standard physics.

Significance and Claims
The paper positions MaD Physics as a stepping stone toward benchmarks that capture the exploratory and unstructured nature of scientific discovery. The authors claim that MaD Physics effectively identifies shortcomings in frontier-level models regarding dynamic information-seeking tasks. Specifically, the benchmark highlights:

• Poor downstream predictive performance when agents cannot rely on memorized laws.
• Difficulty in recovering accurate symbolic expressions for physical laws, even when data is available.
• The necessity of strategic planning under constraints, a capability where current models show significant room for improvement.

The authors conclude that while current models show promise in reasoning, they struggle with the practical realities of scientific inquiry: allocating limited resources to maximize information gain and inferring novel dynamics from constrained, noisy data. Future work is suggested to explore more sophisticated agent scaffolds, optimized prompting, and extensions to more complex physical systems.