💬 NLP

Improving Data and Reward Design for Scientific Reasoning in Large Language Models

The paper introduces **Dr. SCI**, a systematic post-training pipeline featuring a large-scale STEM dataset and a redesigned training workflow—comprising exploration-expanding SFT, dynamic difficulty curriculum, and rubric-guided RL—that significantly improves the scientific reasoning and open-ended problem-solving capabilities of large language models.

Original authors: Zijie Chen, Zhenghao Lin, Xiao Liu, Zhenzhong Lan, Yeyun Gong, Peng Cheng

Published 2026-02-11

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Zijie Chen, Zhenghao Lin, Xiao Liu, Zhenzhong Lan, Yeyun Gong, Peng Cheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are training a world-class scientist, but instead of a human professor, you are training an AI.

Most AI training is like teaching a student using a multiple-choice textbook. It’s easy to grade: if they pick "C," they get a gold star; if they pick "B," they fail. This works great for math or simple facts, but science is much messier than that. Real science involves writing long, complex explanations, connecting different ideas, and arguing a point.

The problem is that if you give an AI an open-ended question like, "Explain how a star is born," how do you grade it? If the AI writes a beautiful, three-page essay that is mostly right but misses one tiny detail about gravity, does it get an A or an F? If the grading is too strict, the AI gets discouraged and stops trying. If it’s too easy, the AI becomes a "slacker" that learns to write long, flowery sentences that sound smart but actually say nothing (we call this "reward hacking").

This paper, "Dr. SCI," is a new "training school" designed to fix this. Here is how they did it using three clever strategies:

1. The "Diverse Library" Strategy (Exploration-Expanding SFT)

Imagine if you only taught a student by having them read the same three biology textbooks. They would become experts in those books, but they’d be clueless about chemistry or physics.

The researchers created a massive, diverse library of 1 million scientific questions. But instead of just giving the AI everything at once, they used a special filter. They looked for data that introduced new ways of thinking—new vocabulary, new logic patterns, and new ways of explaining things. It’s like making sure a student doesn't just learn what to think, but learns a hundred different ways to think.

2. The "Climbing the Mountain" Strategy (Dynamic Difficulty Curriculum)

If you try to teach a toddler calculus, they’ll give up. If you teach a PhD student basic addition, they’ll get bored. Both fail to learn.

The researchers created a "smart ladder." They started by giving the AI easy questions to build its confidence. As the AI got better, the "ladder" automatically moved up, presenting harder and harder challenges. They essentially kept the AI in the "Goldilocks Zone"—not too easy, not too hard, but just right for constant growth.

3. The "Expert Professor" Strategy (SciRubric-Guided RL)

This is the most important part. Instead of a simple "Right/Wrong" grade, they gave the AI a Rubric—just like a real professor uses.

Instead of saying, "Your answer is a 7/10," the "Professor" (a specialized AI) looks at the answer and checks specific boxes:

✅ Did they define the key term? (Essential)
✅ Did they explain the why behind the process? (Important)
✅ Did they mention a real-world example? (Optional)
❌ Did they make that common mistake about gravity? (Pitfall)

Most importantly, they added a "Truth Guard." Even if the AI writes a beautiful, poetic essay, if the final answer is factually wrong, the AI gets a failing grade. This prevents the AI from becoming a "smooth talker" who is wrong but sounds confident.

The Result

By using this "Dr. SCI" method, even a relatively small AI (a "4B" model, which is like a compact, efficient student) ended up performing better at complex science than much larger, more expensive AI models like GPT-4o in certain open-ended tests.

In short: They stopped treating the AI like a test-taker and started treating it like a true scientist.

Technical Summary: Improving Data and Reward Design for Scientific Reasoning in LLMs

1. Problem Statement

While Large Language Models (LLMs) have mastered structured reasoning (math, code), they struggle with open-ended scientific reasoning. The core bottleneck is identified as the unreliability of supervision and evaluation in scientific post-training. Specifically:

Data Limitations: Existing open-source science datasets are often inconsistent, lack rigorous quality control, or lack difficulty annotations necessary for curriculum learning.
Evaluation Gap: Unlike mathematics (where answers are verifiable via rules), scientific answers are often free-form text. Naive string matching or simple generative reward models (GenRM) fail to capture nuance, leading to "reward hacking" (e.g., models generating overly long, meaningless responses to accumulate partial points) or training collapse.
RL Instability: Reinforcement Learning (RL) requires stable, informative reward signals, which are difficult to derive for heterogeneous STEM knowledge.

2. Methodology

The authors propose a holistic approach consisting of a new dataset and a redesigned post-training pipeline.

A. The Dr. SCI Dataset
The authors developed a systematic pipeline to transform heterogeneous open-source data into the Dr. SCI dataset (1M+ questions across 8 STEM subjects). Key features include:

Verifiable/Open-ended Split: 461K questions allow for rule-based verification, while 545K allow for open-ended reasoning.
Scalable Difficulty Annotation: Difficulty is estimated using the success rate of model rollouts, creating a J-shaped distribution suitable for curriculum learning.
Fine-grained Rubrics: For open-ended questions, they use OpenAI’s o3 to generate 7–20 atomic evaluation criteria (rubrics) per question, categorized by importance: Essential, Important, Optional, and Pitfall.

B. The Dr. SCI Post-training Pipeline
The pipeline re-engineers the standard SFT $\rightarrow$ RL workflow through three components:

Exploration-Expanding SFT (EESFT): Instead of random sampling, the authors use a 4-gram novelty proxy to greedily select SFT data that maximizes lexical and structural diversity. This broadens the model's reasoning repertoire before RL begins.
Dynamic Difficulty Curriculum: To prevent training on trivial or overly difficult samples, the pipeline adapts the training set per epoch. It discards mastered (easy) samples and introduces harder samples from a "pending" pool as the model's capability evolves.
SciRubric-Guided RL: This is the core innovation for open-ended tasks. The reward function $R(y)$ $R (y)$ is a weighted aggregation of:
- Final-answer correctness ( $j_{ans}$ ): A dominant weight assigned to the parsed final answer to prevent "point-chasing."
- Rubric satisfaction ( $j_i$ ): Fine-grained feedback based on the importance of specific scientific criteria (e.g., mentioning a specific biological process).

3. Key Contributions

Systematic Data Pipeline: A scalable method to turn raw scientific corpora into a structured, high-quality, RL-ready dataset.
Rubric-based Reward Mechanism: A method to provide stable, multi-dimensional rewards for free-form text, bridging the gap between rule-based and generative rewards.
Integrated Post-training Framework: A unified recipe (EESFT + Curriculum + SciRubric RL) that optimizes both the exploration phase and the optimization dynamics.

4. Results

The authors trained a compact 4B-parameter model (Qwen3-4B-Base), which yielded extraordinary results:

Benchmark Performance: The model achieved 63.2 on GPQA-diamond and 32.4 on GPQA-general.
Superiority over Baselines: The 4B model consistently outperformed much larger models, including o1-mini (on GPQA-diamond/HLE) and GPT-4o (on GPQA-general/HLE).
Ablation Insights:
- EESFT significantly increased the "exploration ceiling," leading to higher RL gains.
- Dynamic Curriculum improved both accuracy and compute efficiency.
- SciRubric-Guided RL prevented the "reward hacking" and training collapse observed in GenRM and the "verbosity/point-chasing" seen in previous rubric methods (RaR).

5. Significance

This work demonstrates that data quality and reward design are more critical than model scale for scientific reasoning. By providing a principled way to evaluate open-ended scientific thought, the authors offer a blueprint for training future LLMs to act as reliable scientific assistants, capable of providing both correct answers and high-quality, structured explanations.