COMPOSITE-Stem

1. Problem Statement

The paper addresses the critical gap in evaluating AI agents for real-world scientific workflows. While frontier AI models have advanced, existing benchmarks suffer from three main limitations:

• Saturation and Contamination: Many expert-written benchmarks (e.g., GPQA, Humanity's Last Exam) have become saturated due to model training on the data, inflating scores without reflecting true reasoning capabilities.
• Static and Constrained Outputs: Traditional benchmarks rely on multiple-choice or exact-match grading, which fails to capture the nuance of open-ended scientific reasoning, intermediate steps, or partial progress.
• Lack of Agentic Realism: Most evaluations do not test agents in executable environments where they must interact with tools, install packages, and process multimodal data (images, documents) as a researcher would.

2. Methodology

The authors introduce COMPOSITE-STEM, a benchmark designed to evaluate AI agents in a terminal-based, executable environment.

A. Task Composition

• Scale & Scope: 70 expert-curated tasks across Physics (20), Chemistry (23), Biology (20), and Mathematics (7).
• Expertise: Tasks were authored by doctoral-level researchers, faculty, and industry practitioners from top global institutions (e.g., MIT, Cambridge, Stanford).
• Multimodal Support: 18 tasks include reference assets (17 images, 1 PDF) mounted into the agent sandbox, requiring visual analysis (e.g., microscopy, X-rays) or document interpretation.
• Environment: Tasks run in a Harbor (Terminal-Bench style) sandbox using a modified Terminus-2 agent harness. The environment provides a Python 3.12 runtime with scientific libraries (RDKit, SciPy, SymPy) and allows web search via installed packages.

B. Evaluation Framework: AsymmetryZero

To overcome the limitations of exact-match grading, the paper utilizes the AsymmetryZero protocol:

• Hybrid Grading: Tasks are graded using a mix of Exact-Match (35 tasks) and Semantic LLM-Jury (34 tasks).
• Rubric-Based Scoring: Each task is defined by a set of criteria (1–40 per task). Criteria can be graded via string extraction or by an LLM jury.
• LLM-as-a-Jury: A panel of five frontier models (DeepSeek-v3.2, GLM-5, GPT-oss-120b, Llama-3.3-70b, Kimi-K2.5) acts as a jury. They vote on whether a submission meets specific semantic criteria.
• Aggregation: A strict majority voting system aggregates jury votes. A task is considered "passed" if it meets the weighted threshold (e.g., 7/10 points on a rubric), allowing for the evaluation of partial correctness.

C. Experimental Setup

• Models Evaluated: Four frontier models were tested: claude-opus-4.6, gemini-3.1-pro, gpt-5.4, and grok-4.20-beta.
• Constraints: Agents operate under a fixed budget of 10 turns and a 3600-second timeout.
• Metrics: Primary metric is Pass@1 (success rate on the first attempt).

3. Key Contributions

1. First Agentic STEM Benchmark with Multimodal Support: Unlike previous static QA benchmarks, COMPOSITE-STEM requires agents to execute code, install dependencies, and interpret visual data within a sandbox.
2. Expert-Driven Curation Loop: The tasks were refined through a multi-round "Portex Datalab" workflow where experts iterated on prompts and rubrics based on real-time model failure analysis, ensuring high discriminative power.
3. AsymmetryZero Grading Protocol: A novel, auditable framework that combines deterministic exact-match checks with flexible, rubric-based LLM judging, enabling the assessment of complex, open-ended scientific reasoning.
4. Open-Source Ecosystem: All 70 tasks, the Harbor adapter, and the grading framework are open-sourced to promote reproducibility and further research.

4. Results

The evaluation reveals significant performance gaps between models and highlights specific failure modes:

• Top Performance: claude-opus-4.6 achieved the highest score at 21.4% Pass@1, followed by gemini-3.1-pro at 18.6%.
• Lower Performance: gpt-5.4 (4.3%) and grok-4.20-beta (5.7%) performed significantly worse.
• Failure Analysis:
  • Solution Errors vs. Submission Errors: Stronger models (Claude, GPT, Grok) mostly failed due to Solution Errors (incorrect answers despite valid submissions). gemini-3.1-pro had a higher rate of Submission Errors (failing to produce an output or hitting turn limits).
  • Tool Usage Correlation: A strong correlation exists between tool usage and success. Models that installed packages (e.g., rdkit) and engaged in deeper exploration (averaging ~10 steps) performed better. Models that relied on "no substantive tool use" had pass rates as low as 3–5%.
  • Grader Disagreement: The jury showed higher entropy (disagreement) on the stronger models (Claude/Gemini), suggesting their answers were often "borderline" or partially correct, whereas weaker models received unanimous failures.

Case Study Insight: In a chemistry task involving SMILES parsing, claude-opus-4.6 succeeded by persistently installing rdkit and using it correctly. gpt-5.4 failed after attempting multiple tool paths and resorting to a fragile, hand-rolled parser that produced an incorrect result.

5. Significance

• Realistic Benchmarking: COMPOSITE-STEM moves beyond "Google-proof" multiple-choice questions to test the actual workflow capabilities of AI agents in scientific domains, including environment management and multimodal reasoning.
• Trust and Adoption: By demonstrating that even top-tier models struggle with these tasks (max 21.4% success), the benchmark provides a realistic baseline for scientists to determine when AI agents are ready for real-world deployment.
• Future Direction: The paper highlights that current agents lack the persistence and tool-integration skills required for complex scientific discovery. The open-sourced nature of the benchmark invites the community to develop agents capable of handling longer-horizon, multi-step scientific problems.

In summary, COMPOSITE-STEM establishes a new standard for agentic evaluation in science, proving that current frontier models have significant room for improvement in executing complex, multimodal scientific workflows.