Technical Summary: TRIAGE – Role-Typed Credit Assignment for Agentic Reinforcement Learning

1. Problem Statement

In Agentic Reinforcement Learning (RL), agents interact with environments through sequences of actions (e.g., searches, clicks, edits, navigation) to achieve a final goal. Standard Group Relative Policy Optimization (GRPO) assigns credit based solely on the final verifier outcome (success or failure). This outcome signal is broadcast uniformly to all action tokens within a trajectory.

The authors identify that this "outcome-only" approach is structurally incomplete for agentic tasks, creating two systematic blind spots:

1. Useful Exploration in Failed Rollouts: Trajectories that fail may contain critical information-gathering actions (e.g., reading a file, running a test) that improved the agent's belief state but did not immediately solve the task. Uniform punishment suppresses these valuable exploratory behaviors.
2. Regression in Successful Rollouts: Trajectories that succeed may contain redundant, harmful, or regressive actions (e.g., wrong edits, repeated clicks) that were later recovered from. Uniform reinforcement incorrectly credits these harmful steps simply because the final outcome was positive.

The core problem is that the final outcome signal cannot distinguish between the semantic role of a local segment (e.g., decisive progress vs. exploration vs. regression) and the global trajectory success.

2. Methodology: TRIAGE

The authors propose TRIAGE, a framework that introduces a semantic role axis to complement the outcome axis. TRIAGE does not replace the verifier but augments the credit assignment process with a structured, role-conditioned correction.

2.1 Role Taxonomy

The framework classifies each environment-facing segment into one of four semantic roles:

• Decisive (D): Actions that produce verifier-checkable progress or complete a necessary subgoal (e.g., buying an item, submitting an answer).
• Exploration (E): Actions that reveal relevant state or improve the information state without immediate task completion (e.g., reading a test, inspecting a container).
• No-Progress (N): Harmless infrastructure actions that change neither task state nor belief state (e.g., empty traversals, redundant clicks).
• Regression (R): Actions that corrupt state or repeat information without gain (e.g., wrong edits, wrong purchases).

2.2 The TRIAGE Algorithm

TRIAGE operates in two stages during the GRPO training loop:

1. Structured Role Judgment: A bounded local context window (e.g., 5 previous and 5 future action-observation pairs) is fed to a structured LLM judge. The judge assigns a discrete role label ($\rho \in \{D, E, N, R\}$) to each segment. Crucially, the judge is not used as an unconstrained reward model; it acts as a classifier to diagnose the local semantic role.
2. Role-Conditioned Credit Assignment: The advantage for a segment $k$ in trajectory $i$ is calculated as:
   $$A^{TRIAGE}_{i,k} = A^{GRPO}_i + \lambda c_{\hat{\rho}_{i,k}}$$
   Where:
   • $A^{GRPO}_i$ is the standard group-normalized trajectory advantage (the optimization direction).
   • $c_{\hat{\rho}}$ is a fixed process reward constant specific to the assigned role (e.g., $c_D=1, c_E=0.5, c_N=-0.1, c_R=-0.5$).
   • $\lambda$ is a hyperparameter controlling the mixing strength of the local signal.

This design ensures the trajectory outcome remains the primary source of optimization direction, while the role labels provide bounded, segment-level corrections to address the blind spots.

2.3 Theoretical Justification

The paper provides a theoretical justification showing that role-conditioned credit is the Mean Squared Error (MSE)-optimal segment correction expressible from role labels alone.

• The "credit residual" (the difference between the true per-segment advantage and the uniform trajectory advantage) is projected onto the role variable.
• If the role labels explain any portion of this residual (i.e., $E[\delta | \rho] \neq 0$), the role-conditioned correction reduces the advantage estimation error compared to uniform broadcasting.
• This reduction in estimation error connects to lower-variance policy gradients, provided the judge is reliable.

3. Key Contributions

1. Identification of Blind Spots: The paper formally defines the two structural failures of outcome-only credit: the suppression of useful exploration in failed rollouts and the reinforcement of regression in successful rollouts.
2. TRIAGE Framework: Introduction of a role-typed credit assignment system that uses a structured LLM judge for semantic diagnosis and fixed role-conditioned rules for credit adjustment, maintaining the GRPO outcome as the base signal.
3. Theoretical Grounding: Proof that role-conditioned credit is the optimal projection of the advantage residual onto role labels, linking reliable role typing to reduced variance in policy gradients.
4. Empirical Validation: Comprehensive evaluation across three agentic benchmarks (ALFWorld, Search-QA, WebShop) demonstrating that gains stem specifically from role typing rather than merely adding dense rewards.

4. Experimental Results

The authors evaluated TRIAGE on ALFWorld (embodied planning), Search-QA (retrieval), and WebShop (e-commerce) using two policy models (Qwen2.5-7B and Qwen3-1.7B).

• Performance Gains: TRIAGE consistently outperformed the standard GRPO baseline.
  • On Qwen2.5-7B, success rates improved by up to 7.9 percentage points (e.g., ALFWorld: 79.6% $\to$ 87.5%).
  • On Qwen3-1.7B, improvements were even more significant, reaching up to 18.4 percentage points.
• Efficiency: TRIAGE reduced the number of environment-facing turns in completed rollouts by 10.4% (ALFWorld) and 14.8% (WebShop) relative to GRPO.
• Ablation Studies:
  • Role Typing vs. Dense Rewards: A scalar process reward baseline (using the same judge but without role typing) improved over GRPO but underperformed TRIAGE. This confirms that the gain comes from the semantic distinction of roles, not just the addition of a dense reward signal.
  • Regression Suppression: The dominant source of improvement was the suppression of regression credit in successful trajectories (removing the "R in success" blind spot).
  • Judge Reliability: The "thinking" capability of the LLM judge was critical. A non-thinking judge failed to detect regression in successful rollouts, causing TRIAGE to underperform GRPO. This highlights that accurate role diagnosis is a prerequisite for the method's success.
• Baselines: TRIAGE outperformed a learned shared-backbone value baseline and a state-grouping method (GiGPO), particularly in WebShop where regressive actions (repeated clicks) are indistinguishable by outcome-trained value heads but clear to a semantic role classifier.

5. Significance and Claims

The paper claims that agentic RL requires a semantic role axis in addition to the outcome axis. The primary significance lies in the distinction that exploration is not "no-progress." By explicitly categorizing actions that improve the agent's belief state (exploration) versus those that corrupt state (regression), TRIAGE enables training in sparse-reward environments without brittle behavior.

The authors emphasize that TRIAGE is a principled path for agents whose success depends on information gathering and recovery. The method does not assume the judge is perfect; rather, it relies on the judge's ability to perform asymmetric error correction (finding useful steps in failures and harmful steps in successes). Theoretical and empirical results suggest that as long as the judge is reliable enough to identify these specific conflict cells, the role-conditioned correction reduces advantage estimation error and leads to more efficient and robust policy learning.