Technical Summary: ARCO (Adaptive Rubric CO-evolution)

Problem Statement

Reinforcement learning (RL) for multi-step Large Language Model (LLM) agents, such as those used in multi-hop question answering (QA), typically relies on scalar rewards. These rewards are often either outcome-based (binary success/failure at the trajectory level) or process-based (opaque scalar scores per step).

• Outcome rewards fail to provide granular credit assignment, making it difficult to distinguish between a lucky guess and a methodical search process.
• Process rewards improve granularity but often remain "black boxes," offering no explanation for why a specific step was good or bad.
• Existing rubric-based methods attempt to improve interpretability by using natural language criteria. However, they suffer from two critical limitations:
  1. Trajectory-level scoring: They assign a single score to the entire rollout, leaving step-level credit assignment unresolved.
  2. Static judges: They rely on frozen, closed-source LLMs (e.g., GPT-4) as external judges. While the rubric text might be updated, the judge's parameters remain static, preventing the scoring function from adapting to the agent's evolving failure modes.

Methodology: ARCO

The authors propose ARCO (Adaptive Rubric CO-evolution), a framework where the evaluation criteria (rubric) and the scoring function co-evolve with the policy at the parameter level using a same-scale open-source model.

1. Hierarchical Rubric Model ($\mu$)

ARCO instantiates both rubric generation and scoring within a single trainable model $\mu$ that shares a backbone $f_\theta$ with the policy $\pi$. The model features two lightweight heads:

• Generation Head (High-level): Given the trajectory prefix $h_t$ and action $a_t$, it generates an ordered set of $K$ natural-language criteria $r_t = (c_{t,1}, \dots, c_{t,K})$. These criteria are context-specific, adapting to the stage of the task (e.g., entity targeting for early searches vs. answer support for final steps).
• Score Head (Low-level): It takes the same state-action context and the generated criteria $r_t$ as input. It produces $K$ dimension-level scores $d_t \in [-1, 1]^K$, which are averaged to form the step-level reward $s_t$.

2. Co-Evolutionary Training

The framework operates on on-policy data, jointly updating the policy $\pi$ and the rubric model $\mu$:

• Supervised Warmup: Before RL, a closed-source teacher annotates trajectories with rubrics and scores. The policy is initialized via Supervised Fine-Tuning (SFT) on successful trajectories, and $\mu$ is initialized to generate criteria and predict scores that satisfy a trajectory-level decomposition constraint.
• RL Loop:
  • Policy Update ($\pi$): The policy is updated using reward-to-go advantages derived from the step-level scores $s_t$ predicted by the current $\mu$. A position-bucketed baseline is used to reduce variance.
  • Rubric Model Update ($\mu$): The model is updated using a trajectory decomposition constraint. The sum of the predicted step scores must equal the terminal binary outcome $R(\tau)$:
    $$\mathcal{L}_{\mu}^{rl} = \sum_{\tau} \left( \sum_{t} s_t - R(\tau) \right)^2 + \beta \cdot \text{KL}(\mu \| \mu_{ref})$$
    This allows $\mu$ to learn step-level credit assignment without gold step labels, while the KL term keeps generated rubrics coherent with the warmup distribution.

3. Key Design Choices

• Shared Backbone: The generation and scoring heads share parameters. This ensures the criteria are written in the same representation space used for scoring, allowing gradients from the decomposition objective to shape features for both stages.
• Same-Scale Open Source: ARCO uses open-source models (e.g., Qwen3-4B, Llama-3.2-3B) for both the agent and the evaluator, eliminating dependence on external closed-source judges.

Key Contributions

1. Adaptive, Step-Level Rubric Framework: ARCO introduces a process reward framework where both the natural-language rubric and the scoring function co-evolve with the policy at the parameter level, removing the need for a frozen closed-source judge.
2. Hierarchical Model with Decomposition: The authors design a model that generates context-specific criteria and scores actions under them. By enforcing a trajectory decomposition constraint, the model learns to assign credit to individual steps based solely on the terminal outcome.
3. Empirical Validation: Extensive experiments on multi-hop QA benchmarks demonstrate that ARCO outperforms strong baselines, and analyses confirm that the generated rubrics are step-specific, robust, and useful for diagnosing agent behavior.

Experimental Results

The authors evaluated ARCO on three multi-hop QA benchmarks: HotpotQA, 2WikiMultiHopQA, and MuSiQue, using two open-source backbones (Qwen3-4B and Llama-3.2-3B).

• Performance: ARCO achieved the best Exact Match (EM) scores in all six (dataset, backbone) configurations and the best F1 scores in five of six. It consistently outperformed:
  • Outcome Reward Models (ORMs): Including binary-reward methods (Search-R1, R1-Searcher) and rubric-based ORMs with frozen judges (RaR, CARMO, RLER).
  • Process Reward Models (PRMs): Specifically AgentPRM, which uses opaque scalar step values.
• Ablation Studies:
  • Removing rubric text generation ("w/o Rubric") caused a significant drop in performance, confirming that rubrics provide essential semantic structure rather than just decoration.
  • Using fixed dataset-level criteria ("w/o Adaptive") performed worse than adaptive generation, indicating that fixed criteria cannot track evolving failure modes.
  • Conditioning rubrics only on the question/action without the trajectory prefix ("w/o Prefix") reduced answer correctness, showing the importance of full context.
• Rubric Quality: Diagnostic tests showed that ARCO's rubrics are highly action-bound (42–55% accuracy in identifying the specific action they scored, significantly above chance) and rated highly for specificity (approx. 4.0/5.0).
• Sensitivity: The number of criteria ($K$) was found to be sensitive; $K=3$ yielded optimal performance, while higher $K$ values introduced redundancy without improving scores.

Significance and Claims

The paper claims that ARCO represents a shift from static, opaque reward modeling to adaptive, interpretable, and co-evolutionary reward systems.

• Interpretability: By generating natural-language criteria for every step, ARCO provides a "readable" signal that explains why an action was rewarded or penalized, aiding in the diagnosis of agent behavior.
• Adaptability: Unlike methods using frozen judges, ARCO's evaluator evolves alongside the policy. As the agent learns to avoid certain errors, the rubric model adapts to identify subtler failure modes (e.g., targeting the wrong entity or redundant retrieval).
• Efficiency and Accessibility: By using same-scale open-source models for both the agent and the evaluator, ARCO avoids the cost and dependency of closed-source APIs while maintaining high performance.

The authors conclude that ARCO offers a foundation for building interpretable process-reward learning systems for language agents, turning sparse outcome rewards into dense, step-aware signals without external judges.