Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy

1. Problem Statement

Despite the critical role of Reward Models (RMs) in Reinforcement Learning from Human Feedback (RLHF), current state-of-the-art open-source RMs exhibit significant limitations:

• Brittleness: They fail to capture the nuanced, sophisticated, and sometimes conflicting spectrum of human preferences.
• Benchmark Saturation vs. Real Performance: Scores on standard benchmarks like RewardBench have saturated but show weak or even inverse correlations with downstream task performance (e.g., Best-of-N scaling or policy training).
• Data Quality Issues: Existing preference datasets are often narrowly scoped, synthetically labeled with low quality, or lack rigorous verification. Simply scaling the quantity of uncurated data does not yield performance gains.
• Stagnation: Many top-performing models rely on similar base architectures or training data, indicating a lack of progress in open preference datasets since late 2024.

The authors hypothesize that the primary bottleneck is not model architecture or training algorithms, but the quality and curation of preference data.

2. Methodology: The Human-AI Synergistic Pipeline

The core contribution is SynPref-40M, a large-scale preference dataset containing 40 million pairs, with 26 million carefully curated pairs used for training. The data is processed via a novel two-stage curation pipeline (Figure 3) that balances human verification quality with AI scalability.

Stage 1: Small-Scale Human-in-the-Loop Curation (Iterative)

This stage focuses on high-quality "Gold" data generation and model refinement through 8 iterative rounds:

1. Seed Initialization: Starts with a small, high-quality seed dataset (Skywork-Reward-Preference-80K-v0.2).
2. Attribute Generation: LLMs generate preference attributes for raw data (task category, objectivity, controversiality, desired attributes, annotation guidelines).
3. Human Verification: Annotators verify labels using a strict protocol, utilizing external tools (search engines, frontier LLMs) for fact-checking and code execution. This produces $D_{gold}$ (verified) and $D_{silver}$ (LLM-verified).
4. Error-Driven Adaptive Retrieval: A reward model is trained on $D_{silver}$ and evaluated on $D_{gold}$. The system identifies pairs where the model performs poorly (low confidence or errors) and retrieves similar unverified pairs from the raw pool for re-annotation.
5. Preference-Aware Labeling: LLMs annotate the retrieved pairs using few-shot examples from $D_{gold}$ to guide their judgment, ensuring alignment with human preferences.
6. Iteration: The process repeats, continuously expanding the dataset and refining the reward model.

Stage 2: Large-Scale Automatic Curation (No Human Supervision)

This stage scales the process to tens of millions of "in-the-wild" pairs ($D_{wild}$):

1. Consistency Filtering: A "Gold Reward Model" (trained on cumulative human-verified data) and the "Best Reward Model" (from Stage 1) jointly evaluate unverified pairs.
2. Selection Logic:
   • Retain: Pairs where the original label agrees with both the Gold RM and the Best RM (or LLM judges).
   • Flip/Recycle: Pairs where the original label disagrees with the Gold RM are "flipped" (swapping chosen/rejected) and added to the training pool, effectively correcting low-quality labels without new annotation.
   • Discard: Pairs with high uncertainty or conflicting signals are discarded.
3. Outcome: This yields a massive, high-quality curated subset (26M pairs) from the initial 40M raw pairs.

3. Key Contributions

1. SynPref-40M: The largest curated preference mixture to date, comprising 40M pairs (26M curated). It demonstrates that high-quality curation is more valuable than raw volume.
2. Skywork-Reward-V2 Series: A suite of eight reward models (ranging from 0.6B to 8B parameters) based on Llama 3.1/3.2 and Qwen3 backbones.
3. Human-AI Synergy Pipeline: A novel framework that uses human verification for quality assurance and LLMs (guided by human attributes and few-shot examples) for scalability. It proves that a small amount of human effort (approx. 2% of the final pool) can drive massive performance gains.
4. Comprehensive Benchmarking: The models are evaluated across seven major benchmarks, including RewardBench v2, PPE, RMB, RM-Bench, and JudgeBench.

4. Experimental Results

The Skywork-Reward-V2 series achieves State-of-the-Art (SOTA) performance across all seven major benchmarks, significantly outperforming existing open models and even larger proprietary models.

• Benchmark Performance:
  • Skywork-Reward-V2-Llama-3.1-8B-40M achieves an average score of 88.6, outperforming the previous SOTA (INF-ORM-Llama3.1-70B, 73.8) by a wide margin.
  • It achieves 97.8 on RewardBench and 86.5 on RewardBench v2.
  • It outperforms generative reward models and LLM-as-a-Judge approaches in most categories, particularly in subjective preference and style resistance.
• Model Size Efficiency:
  • The 1.7B variant outperforms the 70B INF-ORM model on average, proving that data quality can bridge the gap in model capacity.
  • Training on only 1.8% (approx. 290k samples) of the curated mixture already surpasses the previous 70B SOTA.
• Downstream Capabilities:
  • Best-of-N (BoN) Scaling: The models show strong positive scaling in BoN tasks (Helpfulness/Harmlessness), outperforming GPT-4o by up to 20 points.
  • Robustness: They exhibit superior resistance to stylistic biases (RM-Bench) and better factuality scores.
• Ablation Studies:
  • Data Quality > Quantity: Uncurated scaling yields negligible gains; curated scaling yields consistent improvements.
  • Human Guidance is Critical: Removing human guidance from the pipeline (relying only on LLM self-consistency) causes performance to plateau around 74-75%, whereas the full pipeline reaches 83%+.
  • Correction Mechanism: "Recycling" discarded data by flipping labels yields significant performance boosts, validating the pipeline's ability to identify and correct errors.

5. Significance and Impact

• Paradigm Shift: The paper challenges the notion that larger models or complex loss functions are the primary drivers of RM performance. Instead, it highlights data curation quality as the critical factor.
• Cost-Effectiveness: The pipeline demonstrates that a small budget for human annotation (focusing on seed data and verification) combined with automated scaling can produce datasets that outperform massive, expensive human-labeled datasets.
• Open Progress: By releasing high-quality open models and a methodology for data curation, the authors aim to accelerate open-source RLHF research, moving beyond the stagnation observed in late 2024.
• Practical Guidance: The authors provide a "budgeting recipe" for practitioners, showing how to allocate resources between human verification and automated scaling to maximize reward model performance within cost constraints.

In conclusion, Skywork-Reward-V2 represents a substantial leap in open reward modeling, proving that a synergistic human-AI approach to data curation can unlock significantly higher data quality and model performance than previous methods.