Technical Summary: FrontierScience

Problem Statement

As language models rapidly advance in scientific reasoning, existing benchmarks are becoming saturated. Prior evaluations, such as GPQA and MMLU, often rely on multiple-choice formats or knowledge retrieval tasks that do not fully capture the complexity of expert-level scientific reasoning. Furthermore, many current datasets contain pre-existing problems, raising concerns about data contamination and the ability to measure genuine reasoning capabilities versus memorization. There is a critical need for a new generation of benchmarks that are difficult, verifiable, original, and representative of both constrained problem-solving and open-ended research tasks.

Methodology

Benchmark Construction

The authors introduce FrontierScience, a benchmark comprising several hundred questions across physics, chemistry, and biology. The dataset is divided into two distinct tracks:

1. FrontierScience-Olympiad:

   • Content: 160 questions in the open-sourced gold set (derived from over 500 initial candidates), designed to mimic international science olympiad challenges (e.g., IPhO, IChO, IBO).
   • Creation: Problems were authored and verified by 42 former international medalists or national team coaches who collectively hold 108 olympiad medals.
   • Format: Short-answer format requiring precise solutions (numeric or algebraic expressions for physics/chemistry; fuzzy string-matching for biology).
   • Originality: All problems are novel, re-contextualized from existing ideas to prevent contamination, and verified to be unsolvable by current internal models during the creation phase.

2. FrontierScience-Research:

   • Content: 60 questions in the gold set (from over 200 initial candidates), representing sub-problems encountered in authentic PhD-level research.
   • Creation: Authored and verified by 45 domain experts (PhD candidates, post-docs, and professors) from globally recognized institutions.
   • Format: Open-ended tasks requiring judgment and reasoning, estimated to take 3–5 hours for a human expert to complete.
   • Evaluation Architecture: Unlike the Olympiad set, Research tasks are evaluated using a granular, rubric-based system. Each question includes a 10-point rubric with independent, objectively assessable items (e.g., "Writes equation X"). A score of 7/10 is considered a successful solution.
   • Verification: To scale evaluation without human graders, the authors employ a "judge model" (GPT-5 with high reasoning effort) to score model outputs against the rubric.

Data Pipeline

The construction process involves four stages: Creation, Review, Resolution, and Revision.

• Review: Independent experts review each other's tasks. For the Research set, each problem undergoes at least two independent reviews plus a meta-review.
• Selection: Questions that internal models solve correctly (Olympiad) or score highly on (Research) are discarded or significantly modified to ensure difficulty.
• Gold Set: A final meta-review filters the dataset down to the open-sourced gold set, while the remainder is held out to track potential contamination.

Key Contributions

1. Dual-Track Benchmark: The introduction of a benchmark that simultaneously evaluates constrained, precise problem-solving (Olympiad) and open-ended, research-style reasoning (Research).
2. Expert-Authored Originality: The dataset is entirely composed of novel problems created by subject matter experts (medalists and PhD scientists) to ensure high standards of difficulty, originality, and factuality, minimizing data contamination risks.
3. Rubric-Based Evaluation for Research: The paper introduces a novel architecture for evaluating open-ended scientific tasks using detailed, 10-point rubrics. This allows for the assessment of intermediate reasoning steps and failure analysis, rather than relying solely on final answer correctness.
4. Comprehensive Domain Coverage: The benchmark spans a wide array of subfields, from quantum electrodynamics and synthetic organic chemistry to developmental biology and neuroscience.

Results

In initial evaluations of several frontier models (including GPT-4o, o3, o4-mini, GPT-5, GPT-5.1, GPT-5.2, Claude Opus 4.5, Gemini 3 Pro, and Grok 4):

• Top Performer: GPT-5.2 achieved the highest overall scores:
  • Olympiad Set: 77%
  • Research Set: 25%
• Comparative Performance:
  • Gemini 3 Pro scored 76% on the Olympiad set, comparable to GPT-5.2.
  • GPT-5 tied GPT-5.2 on the Research set (25%) and outperformed GPT-5.1.
• Subject Distribution: Models generally performed better on chemistry, followed by physics and biology in the Olympiad set. In the Research set, performance was highest in chemistry, followed by biology and physics.
• Reasoning Effort: Increasing reasoning effort (test-time tokens) significantly improved GPT-5.2's performance, raising its Olympiad score from 67.5% to 77.1% and its Research score from 18% to 25%. Conversely, the o3 model performed marginally worse on the Research set at high reasoning effort compared to medium effort.
• Failure Modes: Analysis of transcripts indicated models typically struggled with reasoning/logic errors, misunderstanding niche concepts, calculation errors, and factual inaccuracies.

Significance and Limitations

The paper claims that FrontierScience provides a more diagnostic view of model strengths and weaknesses for expert-level scientific reasoning than previous benchmarks. It demonstrates that while frontier AI systems have made substantial progress in solving self-contained, constrained problems (Olympiad level), they remain far from saturation on open-ended research-style work (Research level), leaving significant headroom for future improvement.

The authors explicitly acknowledge several limitations:

• Constrained Problem-Solving: The