Technical Summary: ARES (Automated Rubric Synthesis for Scalable LLM Reinforcement Learning)

Problem Statement

Reinforcement Learning with Verifiable Rewards (RLVR) has proven highly effective for improving Large Language Models (LLMs) in domains with automatically checkable outcomes, such as mathematics and coding. However, this paradigm faces a fundamental scalability bottleneck: it relies on binary, outcome-level verification, which is sparse and often infeasible for open-ended tasks like healthcare, instruction following, and professional writing.

Existing attempts to introduce rubric-based rewards (multi-dimensional evaluation criteria) to address this limitation suffer from two primary issues:

1. Scalability: Constructing rubric-annotated data typically requires manual expert curation of questions, reference answers, and evaluation criteria, preventing large-scale deployment.
2. Granularity: Many existing rubrics are defined at the task or dataset level, failing to capture the specific evaluation requirements and nuances of individual questions.

Consequently, there is a lack of scalable pipelines that can generate large-scale, question-specific, weighted rubric data from raw pretraining documents to enable RL training on diverse, open-ended domains.

Methodology: The ARES Framework

The authors propose ARES (Automated Rubric synthEsis for Scalable RL), a six-stage pipeline designed to automatically convert raw pretraining documents into RL-ready instances containing self-contained questions, reference answers, and question-specific weighted rubrics.

1. Data Pipeline

The pipeline operates on raw pretraining documents (e.g., from DCLM, FineWeb-Edu, FinePDFs) through the following stages:

• Document Filtering: An LLM-based classifier removes low-quality, boilerplate-heavy, or context-poor documents unsuitable for question generation.
• Domain & Persona Assignment: Retained documents are annotated with a semantic domain (from 10 categories) and target personas (e.g., Doctor, Student, Caregiver). This conditions the generation to produce diverse question styles and reasoning depths.
• Rubric-Augmented QA Co-Generation: A generator model ($M_{gen}$) simultaneously produces:
  • A self-contained question (answerable without the source document).
  • A reference answer grounded in the source document.
  • A question-specific rubric ($R_q$) consisting of a set of weighted criteria $\{(c_k, w_k)\}$. The rubric includes positive weights for desirable properties and negative weights for common failure modes (e.g., hallucinations, missing critical safety disclosures).
• Quality Verification: The generated QA pairs are validated for self-containment, answer faithfulness to the source, and lack of information leakage (the question does not trivially reveal the answer).
• Rubric Validation: The rubrics are checked for well-formed criteria, coverage of key requirements, valid weight ranges, and sufficient diversity.
• Format Conversion: Validated instances are serialized into a format compatible with Group Relative Policy Optimization (GRPO).

2. Reward Mechanism

Unlike binary rewards, ARES utilizes a rubric-based reward function for RL training. For a question $q$ and response $y$, the reward is computed as:
$$R_{rubric}(q, y; R_q) = \sum_{k=1}^{N_q} w_k^{(q)} \cdot J_\phi(q, y, c_k^{(q)})$$
Where $J_\phi$ is an LLM judge that scores the degree to which the response satisfies criterion $c_k$, and $w_k$ is the importance weight. This provides dense, diagnostic supervision, allowing the model to receive partial credit for correct components even if the overall response is imperfect.

3. Training Setup

The authors employ Group Relative Policy Optimization (GRPO) as the RL optimizer. The policy is trained to maximize the expected reward derived from the rubric-based evaluation, comparing a group of sampled responses to normalize advantages.

Key Contributions

1. ARES Pipeline: A scalable, automated framework that transforms raw pretraining documents into RL instances with self-contained questions, reference answers, and question-specific weighted rubrics.
2. Large-Scale Dataset: The construction of a 100K rubric-annotated dataset spanning ten diverse domains (including Social Science, Medicine, Technology, and Education). This dataset provides fine-grained reward supervision, a capability previously limited to small, manually curated sets.
3. Empirical Validation: Comprehensive evaluation across seven benchmarks demonstrating that rubric-based RL trained with ARES outperforms Continual Pretraining (CPT), Supervised Fine-Tuning (SFT), and binary-reward RL.

Experimental Results

The authors evaluated the ARES-RL approach using Qwen3-4B-Base as the foundation model against four baselines: CPT, NaturalReasoning (SFT), Webscale (binary-reward RL), and ARES-SFT (SFT on generated data).

• Overall Performance: ARES-RL achieved the highest average score across all seven benchmarks, outperforming CPT by 5.33 points, NaturalReasoning by 6.78 points, and binary-reward RL by 4.39 points.
• Open-Ended Tasks: The most significant gains were observed in multi-dimensional, open-ended tasks:
  • Instruction Following (IFEval): +19.27 points over binary-reward RL.
  • Healthcare (HealthBench): +5.37 points over binary-reward RL.
  • Writing (WritingBench): +1.15 points over binary-reward RL.
• Structured Tasks: Despite the dataset containing fewer examples for Math (930) and Coding (1,631) compared to other domains, ARES-RL still achieved state-of-the-art results on GSM8K, HumanEval+, and MBPP+, suggesting that the benefits of rubric-based rewards transfer to structured reasoning tasks.
• Ablation Studies:
  • Structured vs. Holistic: Structured rubric approaches (ARES-RL and General Rubric) outperformed holistic "Blind Judge" baselines, indicating that decomposing evaluation into explicit criteria provides more effective reward signals.
  • Question-Specificity: ARES-RL (question-specific rubrics) outperformed the General Rubric baseline on average, particularly on tasks with varying constraints (e.g., IFEval), highlighting the importance of tailoring criteria to specific questions.
  • Stability: While reference-answer-based rewards performed well on specific benchmarks like HealthBench, they were unstable across diverse task types (e.g., dropping significantly on IFEval), whereas ARES-RL maintained robust performance.

Significance and Claims

The paper claims that ARES addresses the critical bottleneck of scaling RL to open-ended domains by automating the synthesis of high-quality, multi-dimensional reward signals. The authors assert that:

• Richer Supervision: Rubric-based rewards provide denser and more diagnostic learning signals than binary verification or imitation-based SFT, enabling models to optimize for nuanced aspects of response quality (e.g., safety, completeness, constraint satisfaction).
• Scalability: By leveraging raw pretraining documents, ARES eliminates the need for manual expert curation, making rubric-based RL accessible at a scale (100K instances) and domain diversity (10 domains) previously unattainable.
• Generalization: The framework demonstrates that RL optimized with multi-dimensional rubrics can improve performance not only in open-ended domains but also transfer effectively to structured reasoning and coding tasks, even with limited domain-specific data.

The authors conclude that ARES represents a shift from passive token prediction to targeted reward optimization, turning raw documents into focused post-training supervision for complex, multi-dimensional capabilities.