🤖 AI

SCI-PRM: A Tool Aware Process Reward Model for Scientific Reasoning Verification

This paper introduces Sci-PRM, a tool-aware process reward model trained on the newly constructed SCIPRM70K dataset to enhance scientific reasoning by providing fine-grained verification of tool usage and enabling effective test-time scaling and reinforcement learning through dense reward signals.

Original authors: Xiangyu Zhao, Hengyuan Zhao, Yiheng Wang, Wanghan Xu, Yuhao Zhou, Qinglong Cao, Zhiwang Zhou, Lei Bai, Wenlong Zhang, Xiao-Ming Wu

Published 2026-06-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiangyu Zhao, Hengyuan Zhao, Yiheng Wang, Wanghan Xu, Yuhao Zhou, Qinglong Cao, Zhiwang Zhou, Lei Bai, Wenlong Zhang, Xiao-Ming Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a student taking a very difficult science exam. You have a textbook (your knowledge) and a set of specialized calculators and reference books (the tools).

The Problem:
Current AI models are like students who are great at memorizing facts but terrible at using their tools correctly.

If you ask them a simple math question, they might get it right.
But if you ask them to look up a specific chemical reaction in a database or run a complex physics simulation, they often hallucinate. They might pretend they looked up the data when they didn't, or they might write code that looks correct but actually crashes the computer.
Existing "teachers" (AI judges) are good at checking the final answer, but they are bad at watching the student while they work. They can't tell if the student is using the right calculator or if they are just making up the numbers.

The Solution: Sci-PRM
The authors of this paper built a new kind of "super-teacher" called Sci-PRM. Think of it as a strict, highly specialized proctor who sits next to the student and watches every single step of their work in real-time.

Here is how it works, using simple analogies:

1. The "Chain-of-Tool" Training Data

To teach this super-teacher, the researchers didn't just give it questions and answers. They created a massive library of 17,800+ science problems where the "correct" way to solve them involves using tools (like searching the web for a paper or running a code script).

The Analogy: Imagine recording thousands of hours of video where expert scientists solve problems. The video shows not just the final result, but exactly which button they pressed, what code they typed, and how they interpreted the result.
The Twist: They also included "negative examples"—videos of people making mistakes, like writing code that causes an infinite loop or citing a fake scientific paper. This teaches the AI what not to do.

2. The Three-Step "Proctor" Check

When the Sci-PRM looks at a student's work, it doesn't just say "Right" or "Wrong." It acts like a three-part inspector:

Did you pick the right tool? (e.g., Did you use a biology database to look up a protein, or did you accidentally use a math calculator?)
Did you use the tool correctly? (e.g., Did you type the chemical formula correctly, or did you miss a decimal point?)
Did you understand the result? (e.g., The tool gave you a number. Did you interpret it correctly, or did you make up a story that doesn't match the data?)

3. Why It's Better Than Other "Teachers"

The paper compares Sci-PRM to other top-tier AI models (like GPT-5-Mini).

The Gap: When the task is just "thinking" (no tools), the other models are great. But the moment the student has to use a tool, the other models' grades drop significantly. They can't spot the subtle errors in the code or the fake citations.
Sci-PRM's Edge: Sci-PRM stays sharp even when tools are involved. It can spot a "citation hallucination" (a fake paper title) or a "logic error" (code that will crash) without actually having to wait for the computer to run the code.
- Analogy: Other teachers have to wait for the student to finish the whole experiment and see if the beaker explodes before they know it was wrong. Sci-PRM can look at the student's plan and say, "Stop! You're about to mix chemicals that will explode," before anything happens.

4. How It Helps AI Learn (Two Ways)

The paper shows Sci-PRM helps in two specific ways:

Way 1: The "Best of N" Selection (Test-Time Scaling)
Imagine the AI student is allowed to try solving a problem 10 different times.
- Old way: Pick the answer that looks most confident.
- Sci-PRM way: It reviews all 10 attempts, checks every step of the reasoning and tool usage, and picks the one where the student actually followed the rules and didn't make mistakes. This leads to much more accurate answers.
Way 2: The "Coach" for Reinforcement Learning
When training an AI to get better, it needs feedback.
- Old way: The AI tries, fails, and only gets told "Wrong" at the very end. It doesn't know where it went wrong.
- Sci-PRM way: It acts as a dense reward signal. It gives a "thumbs up" or "thumbs down" after every single step. This helps the AI learn much faster and avoid the "vanishing advantage" problem (where the AI gets confused because it doesn't know which specific move caused the failure).

The Bottom Line

The paper claims that Sci-PRM is a specialized tool that makes AI models much more reliable in science. It stops them from making up facts or breaking code when they use external tools. It does this by acting as a step-by-step supervisor that understands both the logic of science and the mechanics of the tools used to solve it.

Key Takeaway: It's not just about getting the right answer; it's about proving you got there the right way, using the right tools, without making things up. Sci-PRM is the first model designed specifically to check that "proof" in the complex world of science.

Technical Summary: Sci-PRM

Problem Statement

While Process Reward Models (PRMs) have demonstrated success in mathematical reasoning, their application to complex scientific domains (biology, chemistry, physics, earth science) remains underexplored. Scientific reasoning differs fundamentally from pure logic; it requires not only logical coherence but also strict adherence to domain knowledge, physical constraints, and factual consistency. Current Large Language Models (LLMs) frequently exhibit domain-specific hallucinations, such as generating plausible but incorrect mechanisms, parameters, or citations.

A critical gap exists in evaluating scientific reasoning trajectories that involve tool use. General-purpose judges (e.g., GPT-5-Mini) maintain reasonable accuracy on standard reasoning steps but degrade significantly when evaluating steps involving scientific tool calls (e.g., code execution for molecular simulation or web searches for literature). Furthermore, existing tool-augmented reward models often focus on generic environments (calculators, simple APIs) and lack the specialized capability to verify scientific workflows, such as checking the validity of a DOI against a paper title or detecting logic errors in code before execution. Additionally, relying on "execution-in-the-loop" for Reinforcement Learning (RL) in scientific discovery is computationally prohibitive due to the high latency of scientific tools (e.g., protein folding simulations can take minutes to hours).

Methodology

1. Dataset Construction: SCIPRM70K

The authors constructed SCIPRM70K, a large-scale dataset featuring "Chain-of-Tool" trajectories.

Sources: Data was aggregated from seven high-quality scientific benchmarks (SGI-DeepResearch, SCI-VerifyBench, Mol-Instruct, MSEarth, ChemBench4K, PHYSICS, etc.) covering biology, chemistry, physics, and earth science.
Trajectory Generation: A generate–execute–judge paradigm was employed. Strong tool-integrated LLMs generated reasoning steps. When a tool call was initiated, the system paused, executed the tool in a real environment (sandbox for code, search engine for web), and appended the actual output to the context before the model resumed reasoning.
Tool Suite: Includes search tools for evidence retrieval and code execution tools using scientific libraries (RDKit, PySCF, Biopython, NumPy, SymPy).
Labeling Pipeline: A two-stage automated labeling process assigns binary labels ( $s_t \in \{0, 1\}$ $s_{t} \in {0, 1}$ ) to each step:
1. Technical Verification: Checks for execution feasibility (e.g., syntax errors, timeouts, empty search results).
2. MCTS Consistency: Uses Monte Carlo Tree Search (MCTS) rollouts to verify if a step consistently leads to the correct gold answer. An LLM judge evaluates tool selection, argument accuracy, and result utilization.
Statistics: The dataset contains 17,818 tool-integrated trajectories and 86,314 annotated steps.

2. Model Architecture: Sci-PRM

Sci-PRM is a tool-aware Process Reward Model trained to provide fine-grained supervision on scientific reasoning.

Backbone: Initialized with Qwen3-VL-8B to leverage multimodal and scientific understanding capabilities.
Training Strategy (Dual-Stage):
1. Supervised Fine-Tuning (SFT): The model is trained to predict both the reasoning process and the step-correctness label simultaneously. The objective focuses on transferring domain-specific grounding knowledge and understanding the "stop-and-go" tool-use protocol.
2. Reinforcement Learning (DAPO-GRPO): To improve robustness and generalization, the model undergoes Dynamic Advantage Policy Optimization (DAPO), a refinement of Group Relative Policy Optimization (GRPO). This stage uses dynamic group sampling and length penalties to encourage efficient scientific inquiry and self-correction.
Function: Sci-PRM assigns a scalar quality score to a reasoning step conditioned on the full history, distinguishing between tool selection errors, calling inaccuracies, and result misinterpretations.

3. Application Modes

Test-Time Scaling (Best-of-N): Given a problem, a base model generates $N$ candidate trajectories. Sci-PRM scores each step, aggregates them into a trajectory score (penalizing length), and selects the most reliable path.
RL Reward Signal: Sci-PRM serves as a dense reward signal for training tool-using agents, mitigating the "advantage disappearance" issue common in sparse outcome-only rewards.

Key Contributions

SCIPRM70K Dataset: A multi-disciplinary dataset with explicit Chain-of-Thought and Chain-of-Tool trajectories, enabling supervised learning of evidence-grounded scientific reasoning.
Fine-Grained Supervision Framework: An automated labeling framework that distinguishes errors in tool selection, calling, and result utilization, providing scalable process supervision beyond mathematics.
Sci-PRM Model: A specialized reward model that scores partial scientific reasoning trajectories, supporting both test-time selection and self-correction to improve reliability and reduce hallucinations.
Empirical Validation: Demonstrated improvements in accuracy and factual consistency across diverse scientific benchmarks compared to Outcome Reward Models (ORMs) and generic tool-use critics.

Experimental Results

Benchmark Performance (SCIPRM-Bench)

Tool-Calling Steps: Sci-PRM achieved a Global Tool F1 score of 0.5619, outperforming the best closed-source baseline (GPT-5-Mini at 0.5338) and open-source baselines (Qwen3-VL-32B at 0.4569).
Non-Tool-Calling Steps: Sci-PRM maintained competitive reasoning capabilities with a Global No-Tool F1 of 0.8333, comparable to GPT-5-Mini (0.8294).
Tool Gap: The performance gap between tool-calling and non-tool-calling steps for Sci-PRM (0.2714) was significantly smaller than for general models like GLM-4.6V (0.3851), indicating better specialization in tool verification.
Search Verification: In detecting hallucinated citations, Sci-PRM achieved the highest Hallucination Detection Rate (HDR) of 0.8820 and an Accuracy of 0.7458, outperforming all baselines.

Downstream Impact

Inference (Test-Time Scaling): Using Sci-PRM for Best-of-N selection on Qwen3-VL-8B improved accuracy across all benchmarks (e.g., ChemBench: 66.17% vs. 60.13% baseline), effectively closing the gap to the theoretical Best-of-N upper bound.
Training (RL): When used as a reward signal for RL, Sci-PRM significantly outperformed Outcome Reward Models (ORMs). Notably, on the Mol-Instructions benchmark, Sci-PRM yielded a +14.39% gain over ORM, attributed to its ability to provide dense feedback on intermediate logical fallacies that ORMs miss.
Efficiency: Sci-PRM achieved an accuracy of 0.76 on tool-calling tasks with an inference time of 0.76 seconds. This contrasts sharply with execution-based approaches (using sandboxes), which achieved 0.86 accuracy but required 282.73 seconds per query due to external API latencies.

Significance

The paper argues that scientific AI agents require a new class of evaluators that are "tool-aware" and capable of verifying the process of reasoning, not just the final answer. Sci-PRM addresses the critical bottleneck of latency in scientific tool execution by internalizing verification capabilities, allowing models to detect code logic flaws and citation hallucinations without expensive runtime execution. By providing dense, step-level rewards, Sci-PRM enables more effective Reinforcement Learning for scientific discovery, helping models break through performance ceilings where outcome-only supervision fails. The work underscores the necessity of fine-grained process supervision for building trustworthy, verifiable, and efficient scientific AI.