Technical Summary: RubricEM

Problem Statement

Training deep research agents—systems capable of autonomous planning, searching, evidence evaluation, and synthesizing long-form reports—presents a fundamental challenge for Reinforcement Learning (RL). Unlike tasks with verifiable ground-truth answers (e.g., math problems or code generation), deep research outputs are open-ended and lack objective correctness metrics. Consequently, standard RL approaches face three critical bottlenecks:

1. Sparse and Delayed Rewards: Feedback is typically coarse and arrives only at the end of long, multi-turn trajectories, making credit assignment difficult.
2. Lack of Reusable Guidance: Conventional post-training methods convert judged attempts into parametric updates but fail to produce explicit, reusable guidance for future attempts.
3. Inefficient Exploration: Without structured intermediate supervision, agents struggle to decompose complex tasks, leading to compounding errors over long horizons.

Existing open-source efforts often rely on verifiable search proxies or high-quality imitation data, while proprietary systems (e.g., Gemini, OpenAI Deep Research) remain opaque regarding their training methodologies. The central question addressed is: How can RL train deep research agents beyond verifiable rewards while enabling long-horizon credit assignment and learning from experience?

Methodology: RubricEM

The authors propose RubricEM, a rubric-guided RL framework that treats rubrics not merely as final-answer evaluators, but as a shared interface structuring policy execution, judge feedback, and agent memory. The framework integrates three core components:

1. Rubric-Guided Policy Decomposition (Structured Scaffold)

Instead of a flat autoregressive process, RubricEM imposes an explicit stage structure on agent trajectories: Plan $\to$ Research $\to$ Review $\to$ Answer.

• Self-Generated Rubrics: During the Plan stage, the agent generates task-specific rubrics (knowledge checklists, analytical criteria, and negative constraints) that guide subsequent stages.
• Operational Rubrics: These rubrics are not static; they condition search queries, serve as references for self-evaluation in the Research and Review stages, and define the criteria for the final synthesis.
• Theoretical Justification: The paper provides a theoretical proof (Theorem 1) demonstrating that explicit stage information strictly improves the expected downstream value of a policy compared to a flat policy when local contexts are ambiguous across different decision modes (e.g., searching vs. synthesizing).

2. Stage-Structured GRPO (SS-GRPO)

To address sparse rewards, RubricEM introduces Stage-Structured Group Relative Policy Optimization (SS-GRPO).

• Dense Credit Assignment: Rather than broadcasting a single terminal score to all tokens, SS-GRPO assigns stage-specific scores ($R_{i,k}$) to Plan, Research, Review, and Answer segments using rubric-guided judges.
• Causal Stage-Dependence: A causal matrix $\Lambda$ defines how downstream stage scores contribute to the return of upstream stages ($G^{\Lambda}_{i,k} = \sum_{j=k}^{K} \lambda_{k,j} R_{i,j}$), ensuring that early stages receive credit for enabling successful downstream outcomes.
• Evolving Rubric Judge: The judge maintains a buffer of discriminative rubrics for each stage, adapting to the policy's current distribution. This allows for process-level feedback without requiring oracle intermediate rewards.

3. Reflection Meta-Policy Training

RubricEM treats experience reuse as an explicit RL objective through a shared-backbone reflection meta-policy.

• Joint Training: The same backbone serves as both the task policy and a reflection generator. After a trajectory is judged, the backbone generates rubric-grounded reflection candidates (summarizing lessons learned).
• Judge-Gated Selection: An LLM judge scores these candidates based on their utility for within-episode refinement (improving the current attempt) and cross-episode transfer (improving future attempts on related questions).
• Rubric Bank: The highest-scoring reflection is stored in a Rubric Bank as natural-language memory. During training, the agent retrieves these reflections to condition future rollouts.
• Asynchronous Efficiency: To avoid sequential bottlenecks, the system employs an asynchronous pipeline where reflection generation and training for step $N$ occur in parallel with task rollouts for step $N+1$, adding negligible wall-clock overhead.

Key Contributions

1. RubricEM Framework: A novel RL architecture that unifies policy decomposition, credit assignment, and memory evolution under a single rubric-guided interface.
2. SS-GRPO Algorithm: A critic-free, stage-aware variant of GRPO that provides denser semantic feedback for long-horizon optimization without requiring step-level oracle rewards.
3. Meta-RL via Reflection: A method for training a shared-backbone meta-policy that distills judged trajectories into reusable, rubric-grounded guidance, effectively turning "judged attempts" into "reusable experience."
4. Empirical Validation: Comprehensive ablation studies and benchmarks demonstrating that the combination of structured scaffolding, stage-wise credit, and reflection training yields superior performance with fewer training steps compared to prior art.

Experimental Results

The authors evaluate RubricEM-8B (an 8B parameter model based on Qwen3) on four long-form research benchmarks: HealthBench, ResearchQA, DeepResearchBench (DRB), and ResearchRubrics.

• Performance: RubricEM-8B achieves a state-of-the-art average score of 55.5 among non-proprietary models, outperforming strong open baselines like DR Tulu-8B (53.6) and WebThinker-32B-DPO (49.0).
• Comparison to Proprietary Systems: It approaches the performance of proprietary systems (e.g., OpenAI Deep Research, Gemini Deep Research), outperforming Perplexity Deep Research on average and surpassing it on the DRB benchmark.
• Efficiency: The model reaches its peak performance in 1,400 RL steps, significantly fewer than the 1,900 steps required by DR Tulu, despite starting from a weaker teacher model (Gemini-3.1-Pro vs. GPT-5).
• Generalization: Despite being trained primarily on long-form data, RubricEM demonstrates strong zero-shot transfer to short-form benchmarks (SimpleQA, 2Wiki, WebWalker, DSQA), suggesting the learned skills are transferable tool-use and evidence-grounding capabilities rather than just report formatting.
• Ablation Studies: Removing any component (structured scaffold, SS-GRPO, or Meta-Policy) results in performance degradation, confirming that all three ingredients are necessary for the observed gains.

Significance and Claims

The paper claims that RubricEM offers a "recipe" for long-horizon RL beyond verifiable rewards: expose task structure, assign credit to that structure, and convert judged attempts into reusable experience.

• Beyond Verifiable Rewards: The work demonstrates that high-quality deep research agents can be trained without ground-truth answers, relying instead on rubric-guided semantic feedback.
• Meta-RL for Agents: It advances the field of Meta-RL for language agents by moving beyond inference-time prompting (e.g., few-shot examples) to training-time evolution, where the agent's parameters and its textual memory co-evolve.
• Scalability: The results suggest that compact models (8B) can achieve competitive performance in complex research tasks when equipped with the correct structural priors and RL algorithms, challenging the notion that only massive proprietary models can perform deep research.

The authors maintain a modest tone, acknowledging limitations such as reliance on external LLM judges (which introduces noise and cost) and infrastructure instability in long-horizon training loops. They posit that future work could explore stronger judges, human-auditable rubric banks, and applications in other domains with multidimensional quality criteria (e.g., scientific review, tutoring).