Reward Modeling for Scientific Writing Evaluation

1. Problem Statement

Scientific writing is a high-stakes, expert-domain task requiring deep domain knowledge, specific reasoning capabilities, and adherence to complex, task-dependent criteria. While Large Language Models (LLMs) have advanced scientific text generation (e.g., generating related work, reviews, or revisions), evaluating these outputs remains a critical bottleneck.

Current evaluation approaches face three main limitations:

• Generalization Failure: Existing LLM-as-a-judge models and reward models are optimized for general benchmarks (e.g., coding, math, safety) with fixed scoring rubrics. They struggle to reason over sparse scientific domain knowledge and fail to adhere to dynamic, task-specific evaluation criteria.
• Cost and Scalability: Fine-tuning a separate evaluator for every specific scientific task is computationally expensive and impractical, especially given the scarcity of annotated scientific evaluation data.
• Reasoning Deficits: Standard models often produce scores without grounding them in explicit reasoning or fail to correct their own errors when presented with contradictory criteria. They often rely on surface-level patterns rather than deep logical alignment with the provided "constitution" (evaluation guidelines).

2. Methodology

The authors propose SciRM (Scientific Reward Model) and SciRM-REF (SciRM with Reflection), a cost-efficient, open-source framework designed to evaluate scientific writing across diverse tasks and aspects.

A. Dataset Construction

The authors curated a multi-task dataset combining:

• Related Work Generation: Evaluated on coherence, positioning type, and positioning consistency (binary scores).
• Review Utility (RevUtil): Evaluated on actionability, grounding, verifiability, and helpfulness (1–5 scale).
• Novelty Assessment & Paper Revision: Added as unseen test tasks to evaluate generalization.
  The dataset includes over 65,000 instances with diverse scoring rubrics, allowing for joint training across multiple evaluation aspects.

B. Two-Stage Training Framework

The core innovation is a two-stage Reinforcement Learning (RL) pipeline using the GRPO (Group Relative Policy Optimization) algorithm:

1. Stage 1: Optimizing Evaluation Preferences (SciRM)

   • Goal: Teach the model to evaluate text based on an explicit, in-context "constitution" (criteria and scoring rubric).
   • Process: The model is prompted with a query, criteria, and examples. It must output a <reasoning> trace followed by a <score>.
   • Reward Function: Rewards are assigned based on formatting compliance and score correctness. A length penalty is applied to prevent reward hacking (generating excessively long or short outputs to game the system).

2. Stage 2: Optimizing Reasoning Abilities (SciRM-REF)

   • Goal: Enhance the model's ability to self-correct and refine its reasoning by "reflecting" on the constitution.
   • Process: The model generates an initial score and reasoning. This output is then fed back into the model with a reflection prompt asking it to re-evaluate the query and criteria before outputting a final score.
   • Reward Function: This stage uses a more complex reward structure that penalizes the model if it changes a correct initial score to an incorrect final one, or rewards it if it corrects an initial error. This encourages the model to stabilize its reasoning and adhere strictly to the rubric.

C. Output Format

To ensure extractability and interpretability, the model is constrained to output structured XML-like tags:

<reasoning> ... </reasoning>
<score> ... </score>

This forces the model to generate a justification before committing to a verdict.

3. Key Contributions

1. Specialized Reward Models (SciRM & SciRM-REF): The first open-source reward models specifically tailored for scientific writing evaluation, capable of handling dynamic criteria and multiple scoring rubrics within a single model.
2. Two-Stage RL Optimization: A novel training regime that separates preference learning from reasoning refinement, significantly improving the model's ability to follow complex, explicit guidelines.
3. Multi-Aspect Joint Training: By training on diverse tasks (related work, reviews, revisions) simultaneously, the models learn transferable evaluation principles rather than overfitting to specific task patterns.
4. Zero-Shot Generalization: The models demonstrate the ability to evaluate unseen tasks (e.g., novelty alignment) and unseen aspects (e.g., grounding in reviews) without task-specific retraining.

4. Experimental Results

The models were evaluated against strong baselines, including open-weight LLMs (Qwen, Llama, Granite), specialized judges (Prometheus, Selene), and proprietary models (GPT-5.2, o3-mini).

• Performance on Seen Tasks: SciRM and SciRM-REF consistently outperformed all baselines in Review Utility and Related Work evaluation. For instance, on the "Positioning Consistency" aspect, the models achieved near-perfect accuracy.
• Generalization to Unseen Tasks:
  • Novelty Alignment: SciRM-REF achieved 74% accuracy, outperforming most open baselines and approaching the performance of the proprietary o3-mini (78%), despite being a smaller, open model.
  • Paper Revision: The models performed comparably to or better than larger proprietary models on instruction-following and correctness tasks.
• Ablation Studies:
  • Removing the reasoning trace generation led to a significant drop in performance (up to 25% on novelty tasks), proving that reasoning is critical for accurate scientific evaluation.
  • The second training stage (Reflection) provided the most significant boost on tasks requiring complex reasoning (Novelty Alignment).
• Efficiency: The models are based on 7B parameter architectures, making them significantly more cost-effective than proprietary 100B+ models while achieving superior or comparable results.

5. Significance and Impact

• Democratizing Scientific Evaluation: By providing open-source, high-performance evaluators, this work reduces the barrier to entry for rigorous scientific text evaluation, moving away from reliance on expensive proprietary APIs.
• Reliability and Interpretability: The requirement for explicit reasoning traces makes the evaluation process transparent. Users can inspect why a score was given, which is crucial for trust in scientific workflows.
• Scalability: The ability to use a single trained model for diverse scientific tasks (reviews, related work, revisions) eliminates the need for expensive, task-specific fine-tuning, addressing the "low-resource" problem in scientific NLP.
• Future Directions: The authors suggest using these reward models to guide the generation of scientific text via RL, creating a feedback loop where models can iteratively improve their writing based on fine-grained, aspect-level feedback.

In summary, this paper presents a robust, generalizable, and cost-effective solution to the "evaluation bottleneck" in scientific AI, demonstrating that specialized reward modeling with structured reasoning can outperform general-purpose judges in expert domains.