Technical Summary: PHYSELITE

Problem Statement

Current benchmarks for evaluating (multimodal) large language models (MLLMs) on physics reasoning suffer from two critical limitations that obscure the true capabilities of frontier models:

1. Insufficient Difficulty and Scale: Existing datasets are often dominated by elementary or K-12 level problems, failing to stress-test complex, multi-step reasoning. While some Olympiad-level datasets exist (e.g., IPhO, APhO), they are limited in scale and updated infrequently, leading to model saturation where performance plateaus before reaching the true ceiling of problem-solving ability.
2. Lack of Process-Level Annotations: Most benchmarks provide only final answers. For physics, the derivation process is as critical as the result. The absence of step-by-step annotations linking visual cues to physical laws prevents the diagnosis of specific reasoning failures and limits the utility of these datasets for training via chain-of-thought supervision or process-level reinforcement learning.

Consequently, high scores on current benchmarks may not represent a model's ability to solve authentic, complex Olympiad-level physics problems.

Methodology

To address these gaps, the authors introduce PHYSELITE, a large-scale, bilingual (Chinese–English), multimodal benchmark designed specifically for Olympiad-level physics reasoning.

Dataset Construction

• Source Material: The dataset comprises 11,586 open-ended problems sourced from the daily training and practice materials of 15 first-prize-winning physics students from local secondary schools. This sourcing strategy aims to provide realistic Olympiad-level difficulty while minimizing pre-training overlap with public competition archives.
• Annotation Pipeline:
  • Multimodal Input: Each problem includes the original text and one or more schematic diagrams.
  • Bilingual Derivations: Problems are paired with human-verified, step-by-step solution derivations in both Chinese and English. Mathematical expressions are converted to LaTeX to preserve precision.
  • Metadata: Each problem is annotated with subject tags (Mechanics, Electromagnetism, Thermodynamics, Optics, Modern Physics) and a difficulty rating on a 1–7 scale.
  • Quality Control: A human-in-the-loop process involving manual verification, OCR correction, and fuzzy matching for deduplication ensures data integrity.

Evaluation Protocol

The authors benchmarked 18 models (8 open-source, 10 closed-source), including both "System-1" (standard) and "System-2" (extended thinking/reasoning) architectures.

• Metrics:
  • Answer Score: Binary correctness of the final result against the reference.
  • Process Score: A step-level evaluation where a model's response is decomposed into key derivation steps. Each step is graded as fully correct (1.0), partially correct (0.5), or incorrect (0.0). This metric allows for partial credit even if the final answer is wrong.
• Judging: Scores are determined by an ensemble of three LLM judges (GPT-5.2, Claude-Opus-4.6, Gemini-3-Pro), validated against human expert grading with a mean absolute error of 0.09.
• Human Baseline: A group of 15 award-winning students solved the problems to establish a human performance baseline.

Key Contributions

1. Scale and Difficulty: PHYSELITE is the largest Olympiad-level physics benchmark to date, containing 11,586 problems—an order of magnitude larger than comparable high-difficulty datasets like PHYBench or HiPhO—while maintaining uniform elite difficulty.
2. Multimodal Process Annotations: Unlike previous datasets that offer only text or final answers, PHYSELITE provides paired visual diagrams and human-verified step-by-step derivations. This supports both rigorous evaluation and process-supervised training.
3. Comprehensive Diagnostic Analysis: The paper provides an extensive evaluation of 18 models, quantifying the performance gap with human experts and characterizing six recurring failure modes across different sub-disciplines and difficulty levels.

Experimental Results

• Performance Gap: Even the strongest model evaluated, Grok-4.2, achieved only 33.7% answer accuracy. This is significantly lower than the human baseline of 48.5%, indicating a substantial gap between current frontier models and expert-level reasoning.
• Open vs. Closed Source: A large performance gap exists between proprietary and open-weight systems. The strongest open-source model (Qwen3-VL-235B-A22B) reached only 20.0% accuracy.
• Reasoning Models: Extended thinking (System-2) models generally outperformed standard models, but the advantage was not uniform. Some strong standard models (e.g., Claude-Opus-4.6) outperformed several reasoning models, suggesting that base capability is a prerequisite for effective extended reasoning.
• Process vs. Answer: Models consistently scored higher on the Process Score than the Answer Score, indicating that models often generate partially valid derivations before failing in the final steps.
• Sub-discipline Weaknesses: Optics emerged as the universal weak point across all models, with significantly higher error rates compared to Thermodynamics. This suggests current MLLMs struggle with geometric and phase-based reasoning.
• Difficulty Trends: Many models exhibited a "U-shaped" performance curve, scoring lower on "Medium" difficulty problems than on "Hard" ones. The authors attribute this to post-training on specific competition-style "Hard" problems, making the "Medium" daily-practice problems out-of-distribution for these models.

Significance and Claims

The paper claims that PHYSELITE serves as a more rigorous and representative benchmark for evaluating the true reasoning capabilities of MLLMs in physics. By moving beyond simple answer matching to process-level evaluation, the authors demonstrate that current models are far from achieving expert-level performance.

The authors emphasize that the dataset is not merely a test suite but a resource for process-supervised training. The detailed step-by-step annotations are intended to facilitate the development of stronger scientific reasoners by allowing models to learn from intermediate reasoning steps rather than just final outcomes. The work highlights that while models have made progress, significant challenges remain in handling complex, multimodal, and multi-step physical reasoning, particularly in domains like Optics and in bridging the gap between standard and reasoning-enhanced architectures.