💬 NLP

Physics-R1: An Audited Olympiad Corpus and Recipe for Visual Physics Reasoning

The paper "Physics-R1" identifies critical flaws in existing multimodal physics evaluation pipelines—specifically data contamination, translation drift, and MCQ saturation—and addresses them by releasing a rigorously audited corpus and a new reasoning model that achieves significant performance gains on novel, open-ended Olympiad benchmarks.

Original authors: Shan Yang

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shan Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of Artificial Intelligence (AI) is like a massive, high-stakes cooking competition. The goal is to see which AI "chefs" can solve complex physics problems (like figuring out how a rocket flies or how electricity moves). To judge them, the competition organizers give them a set of test questions.

The paper you're reading, "Physics-R1," is essentially a whistleblower report and a new recipe book. The author, Shan Yang, argues that the current cooking competition is rigged because the test questions are contaminated, and the judges are biased. Here is the breakdown in simple terms:

1. The Problem: The Test Questions Are "Leaked"

Imagine a student preparing for a math test. If they accidentally memorize the exact answers from a practice book that the teacher also uses for the final exam, they will get a perfect score, but it doesn't mean they actually understand math.

The paper found that many AI models are doing exactly this.

The "Leak": The authors audited the public "training pools" (the practice books) used to teach these AIs. They found that many of the test questions the AIs are being graded on are actually hidden inside the practice books they studied.
The "Paraphrase" Trick: It's not just exact copies. Sometimes the AI sees a question like, "A ball falls from 10 meters," and the test asks, "If a sphere drops from a height of 10m..." The old computer programs used to check for cheating (looking for identical words) missed these.
The New Audit: The authors built a three-stage security guard:
1. The Word Counter: Checks for exact word matches.
2. The Meaning Scanner: Checks if the sentences mean the same thing, even if the words are different.
3. The Human-like Judge: A super-smart AI that reads both the practice question and the test question and decides, "Yes, these are the same problem in disguise."
Using this, they found thousands of "cheating" pairs that the field had missed. They cleaned up the data to create a clean training set (called PHYSCORP-A) where the AI has to actually learn, not just memorize.

2. The Translation Trap: Language Matters

Imagine you are taking a driving test. You pass easily when the instructions are in your native language, but when the same test is translated into a foreign language, you fail because the grammar is confusing or the road signs are described differently.

The authors tested this with physics problems in Estonian (the original language) and English (a translation).

The Result: A top-tier AI (Sonnet 4.5) got 30.5% correct on the original Estonian problems but only 13.6% on the English translations.
The Lesson: Translating complex science problems often loses subtle details. If we only test AI in English, we might think they are worse at physics than they actually are, or we might be testing their translation skills instead of their physics skills.

3. The Format Trap: Multiple Choice vs. Real Thinking

Imagine a student who is great at guessing on a multiple-choice quiz (A, B, C, or D) but freezes when asked to write out the solution on a blank piece of paper.

The paper found a massive gap in how AI performs based on the format of the test:

Multiple Choice (The Easy Mode): The AI scored 79.7% on a multiple-choice test.
Open-Ended (The Hard Mode): The same AI scored only 33.4% on a test where it had to write the answer from scratch.
The Gap: That's a 46-point difference. The paper argues that the field has been overestimating AI intelligence because it's mostly testing them on the "easy mode" (multiple choice) where they can guess or recognize patterns, rather than the "hard mode" (solving from scratch).

4. The Solution: A New Recipe (Physics-R1)

To fix these problems, the authors didn't just point fingers; they built a new kitchen.

The Clean Ingredients: They released a new, audited dataset (PHYSCORP-A) that is guaranteed to be free of the "leaked" questions.
The New Test: They created a new, harder exam called PHYSOLYM-A. It uses original problems (mostly from Estonia and international Olympiads) that the AI has never seen before.
The Cooking Method (Physics-R1): They trained a new AI model using a specific "recipe" (a method called GSPO+DAPO).
- Instead of giving the AI a complex, detailed score for every tiny step (which can trick the AI into writing fancy-looking but wrong answers), they used a Binary Reward.
- The Analogy: Think of it like a referee in a game. Instead of giving points for "good posture" or "nice handwriting," the referee just says: "Did you get the right answer? Yes = 1 point. No = 0 points."
- This simple rule forced the AI to stop trying to "game the system" with fancy formatting and actually focus on solving the physics problem correctly.

The Results

When they tested their new AI (Physics-R1) on the new, clean, hard exam:

It improved significantly compared to the base model (going from an 8% score to a 26% score).
It beat other open-source models.
It still trails behind the very best closed-source "super-AI" (Sonnet 4.5), but it proves that with clean data and the right training method, open-source models can learn real physics reasoning.

Summary

This paper is a call to action for the AI community:

Stop cheating: Clean your training data so models aren't just memorizing test answers.
Watch the language: Don't assume translations are perfect; test in the original language.
Stop guessing: Test models on open-ended problems, not just multiple-choice.
Keep it simple: Sometimes, a simple "Right/Wrong" reward is better than a complex scoring system for teaching AI to think.

The authors have released all their "ingredients" (data), "recipes" (code), and "exam questions" (benchmarks) for anyone to use and verify.

Technical Summary: Physics-R1: An Audited Olympiad Corpus and Recipe for Visual Physics Reasoning

Problem Statement

The field of multimodal physics reasoning relies heavily on vision-language benchmarks to evaluate model capabilities. However, the construction of these benchmarks is rarely audited, leading to three critical methodological gaps that distort cross-model comparisons and inflate frontier rankings:

Train-Eval Contamination: Public training pools (e.g., SciInstruct, UGPhysics-Train) often contain paraphrased or near-duplicate problems relative to evaluation sets. Conventional single-stage $n$ -gram audits fail to detect these "rephrasing dark matter" instances.
Translation Drift: Multilingual benchmarks often rely on English translations of problems originally composed in other languages. These translations can degrade information content (vocabulary, grammatical case, scope), leading to systematic underestimation of model ability on the translated versions compared to the originals.
Format and Novelty Saturation: Current benchmarks are often saturated by Multiple Choice Questions (MCQs) or lack novelty, causing performance to plateau at a "closed-frontier ceiling" where format and task familiarity, rather than genuine reasoning, drive scores.

Methodology

The paper proposes an end-to-end audit of the evaluation pipeline and introduces a corresponding training recipe and corpus to address these gaps.

1. The Three-Stage Audit Pipeline

To detect contamination that evades standard $n$ -gram checks, the authors implement a three-stage audit protocol applied pairwise between training pools and evaluation sets:

Stage 1 (N-gram): Computes 5-gram Jaccard similarity ( $J \ge 0.4$ ) to catch verbatim or near-verbatim duplicates.
Stage 2 (Embedding): Uses mxbai-embed-large to compute cosine similarity ( $\cos \ge 0.85$ ) to catch paraphrased duplicates that Stage 1 misses.
Stage 3 (LLM-Judge): A Haiku-4.5 LLM judge classifies Stage 2 candidates as either close duplicates (paraphrases/numeric variations of the same problem) or same-topic neighbors (related physics but distinct setups). Only close duplicates are removed.

2. Data Construction

PHYSCORP-A: A 6,432-record multimodal corpus constructed by applying the three-stage audit to a 14,294-record pre-audit pool. It includes sources like the Estonian Physics Olympiad (with native difficulty labels), Kevin Zhou's handouts, and seven international olympiads.
PHYSR1CORP: A 2,268-record closed-form RL training pool (MCQ and numerical only) derived from PHYSCORP-A, strictly audited against six public physics evaluations.
PHYSOLYM-A: A 500-problem held-out Olympiad evaluation set. It is 99.8% novel-source (one overlap with OlympiadBench-Physics was disclosed rather than dropped) and includes a bilingual English/Estonian subset for cross-lingual analysis.

3. Physics-R1 Recipe

Physics-R1 is a reference Reinforcement Learning (RL) recipe using GSPO (Group Sequence Policy Optimization) and DAPO (Decoupled Clipping and Dynamic Sampling), cold-started from Qwen3-VL-8B-Thinking.

Reward Function: The authors recommend a binary correctness reward ( $r \in \{0, 1\}$ ) based on matching the extracted boxed answer to the gold solution. They argue this is variance-optimal under GSPO with group-normalized advantages and robust against "Goodhart" channels where models optimize for formatting proxies (e.g., LaTeX structure) rather than physics correctness. A dense, five-component reward (including units, conservation laws, and symbolic form) is reported as an ablation.
Curriculum: Uses an MM-Eureka-style difficulty curriculum, dropping prompts where the base model achieves 0/N or N/N rollouts.

Key Findings

The audit and experiments reveal three primary findings:

Hidden Contamination: While public training pools pass single-stage $n$ -gram audits (0 hits), the three-stage audit reveals significant contamination. For example, in SciInstruct, the audit surfaced 4,846 paraphrase candidates, of which 134 were confirmed close duplicates by the LLM judge.
Translation Delta: On 59 paired Estonian/English Olympiad problems, Claude Sonnet 4.5 achieved 30.5% strict accuracy on the original Estonian text but only 13.6% on the English translation. This 17-point delta suggests that English translations systematically underestimate model capability for models with strong cross-lingual transfer.
Format-and-Novelty Gradient: On identical Sonnet 4.5 weights, performance drops by 46 points across benchmarks: 79.7% on PhyX (MCQ) $\to$ 50.4% on OlympiadBench-Physics (open-ended) $\to$ 33.4% on the audited PHYSOLYM-A (open-ended, novel-source). This gradient demonstrates that current benchmarks are saturated by format and familiarity rather than reasoning depth.

Results

Physics-R1, trained on the audited PHYSR1CORP with a binary reward, demonstrates the trainability of the cleaned corpus:

PHYSOLYM-A: Lifts the 8B base model from 8.0% to 26.3% (3-seed mean), a +18.3 point improvement. It trails Sonnet 4.5 by 7.1 points but outperforms all open-source baselines (including Qwen3-VL-32B) and non-Sonnet closed APIs.
PhysReason: Improves from 23.9% to 39.6%, surpassing Qwen3-VL-32B and Gemini 2.5 Pro.
OlympiadBench-Physics: Improves from 46.2% to 46.2% (note: the abstract states +6.9 pp lift, implying a move from ~39.3% to 46.2% on the specific split used, though the table shows 46.2±1.5 for the 3-seed mean).
PhyX (MCQ): Improves from 77.8% to 77.8% (stable, as MCQs are often saturated).

The paper notes that the binary reward outperforms the dense reward on open-ended tasks (+8.9 pp on PhysReason, +6.4 pp on PHYSOLYM-A) while performing comparably on saturated MCQs.

Significance and Claims

The paper positions itself as a methodological intervention rather than a purely algorithmic breakthrough. Its primary contributions are:

Evidence of Trainability: By showing that a model trained on a rigorously audited corpus improves significantly on a held-out, contamination-free evaluation (PHYSOLYM-A), the authors validate that previous stagnation in physics reasoning benchmarks was partly due to data contamination and saturation, not just model limitations.
Measurement Protocol: The paper argues that defensible physics-VL measurement requires three properties: a three-stage audit, original-language gold standards, and open-ended novel-source evaluation.
Artifact Release: The release of PHYSCORP-A, PHYSR1CORP, PHYSOLYM-A, and the Physics-R1 recipe provides the community with tools to reproduce these findings and build future models on clean data.

The authors modestly claim that Physics-R1 is a "reference recipe" and that the binary reward is the "deployable default," while acknowledging that the dense reward and specific seed sensitivities require further ablation. The work does not claim to solve all physics reasoning challenges (e.g., graduate-level perturbation theory remains difficult) but establishes a cleaner baseline for measuring progress.