Technical Summary: RICE-PO

Problem Statement

The paper addresses the credit-assignment challenge in training language agents for reasoning-intensive retrieval. Unlike traditional one-shot retrieval, modern agents engage in iterative loops where they reason over evidence, reformulate queries, and search again. A fundamental asymmetry exists in this process:

• Executable Actions: Generated summaries or queries can be directly submitted to a retriever and evaluated using standard metrics (e.g., NDCG@10).
• Latent Reasoning Actions: Internal reasoning steps (e.g., thought traces) are not directly executable or measurable. They only influence retrieval indirectly by shaping subsequent executable actions.

Existing Reinforcement Learning (RL) methods often rely on outcome-level rewards (the final retrieval score) assigned to the entire trajectory. This approach is unreliable because a successful final outcome may result from later recovery steps rather than the initial reasoning, leading to credit misassignment. Furthermore, many methods rely on learned critics or process reward models, which are computationally expensive and difficult to train. The core problem is how to assign credit to latent reasoning steps without external judges or exhaustive rollouts, leveraging only the interaction signals available within the retrieval loop.

Methodology: RICE-PO

The authors propose RICE-PO (Retrieval-Interaction Credit Estimation for Policy Optimization), a critic-free framework that converts retrieval interactions into localized learning signals. The method operates on the premise that executable summaries provide direct feedback, which can be selectively propagated to preceding latent reasoning spans under specific conditions.

The framework consists of three main stages:

1. Uncertainty-Guided Anchor Selection

Instead of evaluating every step, the agent identifies critical decision points where credit assignment is most needed.

• Mechanism: The system calculates the token-level entropy of executable summary actions across multiple sampled trajectories.
• Selection: It selects the top-$K$ summary actions with the highest entropy (high uncertainty). These serve as "anchors" for local counterfactual estimation, focusing computational resources on ambiguous decision points where the policy's choice significantly impacts the outcome.

2. Local Counterfactual Branching

For each selected anchor point, the method constructs local branches to estimate the influence of the reasoning action on the retrieval outcome.

• Branch Construction: From the same interaction history, the agent generates $K$ alternative reasoning spans ($z_t$) and their corresponding summary actions ($s_t$).
• Influence Estimation: It computes the variance of retrieval rewards ($\hat{\sigma}^2_t$) across these branches. A high variance indicates that the latent reasoning action strongly influences the quality of the subsequent executable summary.
• Residual Stability Estimation: The branches are extended to the full trajectory depth to compute the final reward. The variance of the residual effect ($\text{ResVar}_t = \text{Var}(R_T - r_t)$) is calculated. A low variance indicates that later interaction steps do not overturn the local reward difference established by the current reasoning-summary pair.

3. Gated Credit Propagation

Credit is propagated from the summary to the reasoning span only when the signal is deemed reliable.

• Gating Conditions: Propagation occurs if:
  1. Influence is High: $\hat{\sigma}^2_t \geq \tau_{var}$ (the reasoning choice significantly alters the summary's retrieval score).
  2. Residual Stability is High: $\text{ResVar}_t \leq \tau_{res}$ (the local reward difference persists through future steps).
• Advantage Calculation:
  • Executable Tokens: Receive a step-level advantage ($A^{sum}_t$) based on the local counterfactual retrieval scores.
  • Reasoning Tokens: Receive a gated advantage ($A^{think}_t$). If the gating conditions are met, they inherit the summary advantage; otherwise, they retain the standard trajectory-level advantage.
• Optimization: These token-level advantages are used in a PPO-style clipped objective to update the policy, eliminating the need for a learned critic or external reward model.

Key Contributions

1. Observability-Aware Credit Assignment: The paper introduces a perspective that distinguishes between executable and latent actions in retrieval, showing that executable summaries can serve as direct feedback anchors while latent reasoning requires selective backward propagation.
2. Critic-Free Framework: RICE-PO exploits the asymmetry of retrieval agents to derive fine-grained supervision without learned critics, process reward models, or human annotations. It uses uncertainty-triggered counterfactual evaluation and residual-stability gating to ensure credit reliability.
3. Empirical Validation: Extensive experiments demonstrate consistent improvements over prompt-based agents and group-based RL baselines (GRPO, GiGPO, HGPO, Tree-GRPO) on reasoning-intensive and general retrieval tasks.

Experimental Results

The method was evaluated on BRIGHT (12 reasoning-intensive tasks) and BEIR (5 general retrieval datasets) using NDCG@10 as the metric.

• BRIGHT Performance:

  • Using a 1.5B backbone (DeepSeek-R1-Distill-Qwen-1.5B), RICE-PO achieved an average score of 27.49, outperforming GRPO (24.98) and the state-of-the-art prompting baseline Diver (37.20 average with a 14B model, though RICE-PO with 1.5B outperformed the 1.5B GRPO baseline significantly).
  • Using a 4B backbone (Qwen3-4B-Thinking-2507), RICE-PO achieved the best average score of 37.76, surpassing GRPO, GiGPO, HGPO, and Tree-GRPO by margins ranging from 1.15 to 1.45 points.
  • The results indicate that learning from interaction signals allows smaller models to compete with or surpass larger models relying solely on fixed prompting or outcome-level rewards.

• BEIR Performance:

  • RICE-PO achieved the best average scores on BEIR datasets for both 1.5B (65.66) and 3B (65.68) backbones, outperforming all RL baselines and strong reranking models.
  • While gains were smaller than on BRIGHT (due to BEIR's lower reasoning intensity), the consistency suggests the method generalizes beyond multi-step reasoning settings.

• Ablation Studies:

  • Gating Mechanism: Dynamically gating credit propagation based on influence and residual stability outperformed strategies that always propagate or rely solely on final rewards.
  • Triggering: Entropy-based triggering for anchor selection yielded significant gains over random triggering across most tasks.
  • Signal Combination: Combining "influence" (local sensitivity) and "effect" (downstream stability) proved superior to using either signal alone.

Significance and Claims

The paper claims that the structure of agent-environment interaction itself can provide useful supervision for training reasoning-based retrieval agents. By treating retrieval interactions as a source of localized learning signals rather than relying solely on sparse, outcome-level rewards, RICE-PO enables more efficient and reliable training of reasoning agents.

The authors emphasize that this approach offers a practical alternative to training expensive critics or relying on larger models. The work highlights the importance of designing learning algorithms that align with the specific structure of multi-step language system interactions, particularly the distinction between observable executable actions and latent reasoning processes. The results suggest that effectively utilizing interaction signals is key to improving reasoning-intensive retrieval, even with smaller language models.