InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis

1. Problem Statement

The paper addresses the critical challenge of evaluating Large Language Models (LLMs) as scientific assistants, specifically their ability to reason from empirical data. Existing benchmarks suffer from several methodological flaws:

• Publication Bias: Real-world datasets are skewed toward positive, "clean" findings, failing to represent scenarios where data is insufficient to draw a conclusion.
• Known-Knowledge Bias: Models often rely on parametric priors (pre-trained knowledge) rather than the provided data, leading to hallucinations or ignoring novel/counterintuitive evidence.
• Label Noise: Human annotations in real-data benchmarks introduce ambiguity and ground-truth errors.
• Scalability & Storage: Distributing large, representative scientific corpora is costly and legally constrained.
• Lack of Unanswerability: Current benchmarks rarely test a model's ability to recognize when a question is unanswerable due to missing data or confounders.

2. Methodology: InfiniteScienceGym

The authors propose InfiniteScienceGym, a procedurally generated benchmark that creates an unbounded number of self-contained scientific repositories paired with verifiable question-answering (QA) tasks. The system operates via a three-stage pipeline:

A. The Simulator (Repository Generation)

Given a random seed, the simulator deterministically generates a realistic file-system structure without storing static data.

1. Scientific Context: Samples a field, domain, and subdomain from a hierarchical taxonomy (22 fields, 244 domains, 780 subdomains).
2. Project Specification: Uses an LLM (Qwen3 4B Instruct) to generate a project title, abstract, and a detailed "data-generating program" (hypotheses, variables, confounders). This specification is privileged (hidden from the evaluated model).
3. Directory Structure: Generates a plausible folder/file tree where paths encode experimental conditions (e.g., dates, variable values).
4. File Structure: Populates files with tabular data. Independent variables are sampled from specific distributions (Bernoulli, Normal, etc.), and dependent variables are generated via Python functions that define relationships (linear, non-linear, noise) based on the independent variables.

B. The QA Generator (Privileged Access)

Using the "privileged" knowledge of the data-generating process, this module creates questions with exact ground truth.

• Answerable Questions: Queries where the answer can be computed from the repository (e.g., statistical aggregation, filtering).
• Unanswerable Questions: Constructed to be impossible to answer due to missing variables, insufficient statistical power, or ambiguous definitions. The ground truth for these is explicitly "not possible."

C. The Paraphrase Module

To ensure robustness against surface-form variations, templated questions are converted into naturalistic research queries using LLMs, mimicking how a human researcher would ask the question.

3. Key Contributions

1. InfiniteScale Benchmark: A seed-driven system that generates unlimited, reproducible scientific repositories and QA pairs without the storage burden of static datasets.
2. Guaranteed Ground Truth & Unanswerability: Unlike real-data benchmarks, the system guarantees the existence (or non-existence) of an answer by construction, allowing for precise measurement of abstention (knowing when not to answer).
3. Agentic Evaluation Framework: Models are evaluated in a tool-enabled environment where they must navigate directories, read files, and use code interpreters (Python) to analyze data, rather than relying solely on context window stuffing.
4. Complementary Perspective: Designed to expose "blind spots" (e.g., failure to abstain, inability to handle unanswerable queries) that are difficult to isolate using published datasets.

4. Experimental Results

The authors evaluated proprietary models (GPT-5.4, Claude Opus 4.6) and open-weight models (GPT-OSS 20B, Gemma 3 27B, Qwen3 4B) on 500 sampled questions.

• Overall Performance: No model achieved high accuracy. The best performer (GPT-5.4) reached only 44.8%. Proprietary models outperformed open-weight models by at least 6.4 percentage points.
• Unanswerable Questions: Recognizing unanswerable questions is a major weakness.
  • Proprietary models showed balanced precision and recall (~80%).
  • Open-weight models exhibited high precision but low recall, meaning they frequently guessed confidently when the answer was impossible, rather than abstaining.
• Strategy Analysis (Tokens vs. Tools):
  • Token Usage: Higher token consumption did not correlate with better performance. In fact, the best model (GPT-5.4) used the fewest tokens (~24k per question).
  • Tool Usage: Accuracy positively correlated with the number of tool calls.
  • Key Insight: Stronger models used programmatic analysis (loading data into a Python interpreter to compute answers) rather than context stuffing (loading massive files directly into the LLM's context window). Weaker models tended to load entire files into context, leading to errors and higher token costs.

5. Significance

• Redefining Evaluation: InfiniteScienceGym shifts the focus from "knowledge retrieval" to "evidence-grounded reasoning." It proves that current SOTA models struggle significantly with the fundamental scientific task of determining if data supports a conclusion.
• Efficiency vs. Brute Force: The results challenge the notion that "more compute/tokens" equals better reasoning. Instead, effective scientific agents must be efficient, using tools to process data programmatically.
• Future Research: The framework provides a controlled environment to study hallucination (by checking if a model accesses data it shouldn't need), bias (by injecting contradictory data), and uncertainty quantification.
• Reproducibility: The seed-based generation ensures that any benchmark instance can be exactly reproduced, facilitating rigorous, fair comparisons between models.

In conclusion, InfiniteScienceGym reveals that while LLMs are improving, they currently lack the robustness to act as reliable scientific assistants, particularly in recognizing data insufficiency and utilizing tools efficiently. The benchmark serves as a necessary "unit test" for the next generation of agentic AI systems.