Technical Summary: Self-Induced Outcome Potential (SIOP)

Problem Statement

Long-horizon Large Language Model (LLM) agents, particularly those engaged in search-augmented reasoning or tool use, succeed or fail based on intermediate interaction turns. However, standard training paradigms typically rely on outcome-level supervision, where rewards are assigned only after the final answer is generated. This creates a supervision mismatch: a single terminal reward cannot reliably attribute credit to specific intermediate thoughts, tool calls, or observations that improved the agent's future prospects.

Existing solutions face two primary limitations:

1. Turn-level shaping methods (e.g., IGPO, TIPS) require access to a "gold answer" or a stable, task-specific verifier to compute potential-based rewards. Such supervision is often unavailable in open-ended agentic settings.
2. Label-free Reinforcement Learning (RL) methods (e.g., TTRL, EMPO) extract signals from output distributions but operate primarily at the trajectory or answer level. They identify reliable final responses but fail to assign value to the intermediate turns that led to those outcomes.

The core challenge is to construct a verifier-free, turn-level credit assignment mechanism that leverages self-generated signals without assuming the existence of a ground-truth answer during training.

Methodology: Self-Induced Outcome Potential (SIOP)

SIOP addresses this by treating semantic clusters of final answers as latent future outcome states. Instead of relying on a fixed gold target, the framework constructs a query-specific target distribution over these latent states and rewards turns based on their ability to increase support for reliable future modes.

1. Semantic Outcome Modes and Latent States

For a given query $q$, the policy samples $K$ rollouts. The final answers $\{y^{(k)}\}$ are clustered into semantic outcome modes ($C(q)$) based on bidirectional semantic entailment (using an NLI model). Each cluster $c$ represents a distinct semantic hypothesis.

• Latent Future States: These clusters serve as the future state space for potential-based shaping.
• Reliability Calibration: Raw cluster frequency ($m(c|q)$) is insufficient, as a frequent cluster may be consistently wrong due to model bias. SIOP introduces a reliability signal $r(c, q)$ (e.g., evidence support from retrieved tool observations) to reweight the empirical distribution.
• Target Distribution: The self-induced target distribution $q_\theta(c|q)$ is formed by calibrating empirical mass with reliability:
  $$q_\theta(c|q) \propto m(c|q) \exp(\eta \cdot r(c, q))$$
  This ensures that frequent but unreliable modes are down-weighted, while less frequent but evidence-supported modes are up-weighted.

2. Multi-Reference Cluster Support

To avoid sensitivity to specific wording, SIOP does not score a single prototype answer. Instead, it selects a small set of reference answers $R(c)$ from each cluster with normalized weights $\alpha_y$. The support for a cluster $c$ given a prefix $\tau_{\le t}$ is approximated as:
$$\tilde{p}_\theta(c|\tau_{\le t}) = \sum_{y \in R(c)} \alpha_y p_\theta(y|\tau_{\le t})$$
This multi-reference approach provides a robust measure of support for the semantic mode as a whole.

3. Potential-Based Turn-Level Rewards

The core innovation is the Outcome Potential $\Phi_\theta$, defined for a rollout's realized mode $c^{(k)}$ at turn $t$:
$$\Phi_\theta(\tau_{\le t}^{(k)}, c^{(k)}) = q_\theta(c^{(k)}|q) \log \tilde{p}_\theta(c^{(k)}|\tau_{\le t}^{(k)})$$
The process reward for turn $t$ is the difference in potential between the current and previous prefix:
$$r_{t,proc}^{(k)} = \Phi_\theta(\tau_{\le t}^{(k)}, c^{(k)}) - \Phi_\theta(\tau_{\le t-1}^{(k)}, c^{(k)})$$
This reward is positive if the turn increases the model's support for the cluster it eventually lands in, weighted by the cluster's calibrated reliability.

4. Turn-Level Advantage Construction

To integrate these rewards into policy optimization:

• Augmented Immediate Reward: A mixing coefficient $\lambda$ combines the process reward with a terminal outcome mass added only at the final answer turn.
• Group-Normalized Advantages: Rewards are pooled within a query group (all rollouts for one prompt), normalized (z-score), and propagated backward via a discounted sum to compute turn-level advantages $A_t^{(k)}$.
• Token-Level Assignment: Unlike standard GRPO which broadcasts a trajectory-level score, SIOP assigns the specific turn advantage $A_t^{(k)}$ only to the trainable tokens generated within that specific turn.

Key Contributions

1. Verifier-Free Turn-Level Credit Assignment: The paper formulates credit assignment as potential-based learning over self-induced semantic outcome modes, eliminating the need for gold answers or task-specific verifiers.
2. Self-Induced Outcome Potential (SIOP): A novel reward function that calibrates empirical outcome modes with reliability signals and converts prefix-to-prefix changes in cluster support into turn-level process rewards.
3. Granular Advantage Construction: The method constructs turn-specific advantages for long-horizon agents, assigning credit only to trainable tokens within each turn, avoiding the "broadcast" of trajectory-level scores that obscures intermediate contributions.
4. Theoretical Characterization: The authors prove that SIOP reduces to standard gold-answer shaping in the supervised limit (where the target distribution concentrates on the gold cluster) and demonstrate that it measures step-wise marginal support gain rather than broadcasting a global score.

Experimental Results

The method was evaluated on seven search-augmented QA benchmarks (NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle) using Qwen3-4B and Qwen3-8B backbones.

• Performance: SIOP consistently outperformed verifier-free baselines (TTRL, EMPO) that rely on outcome-level signals. On the 4B backbone, SIOP achieved the best average verifier-free Exact Match (EM) and F1, improving over EMPO by +0.7% EM and +1.8% F1.
• Comparison to Supervised Baselines: SIOP approached the performance of gold-supervised outcome baselines (GRPO) and significantly narrowed the gap to gold-supervised turn-level baselines (IGPO) without using any gold labels during training. Notably, on multi-hop benchmarks like HotpotQA and 2Wiki, SIOP outperformed the gold-supervised GRPO outcome baseline.
• Ablation Studies: Removing reliability calibration, multi-reference support, or the per-turn process reward all led to performance degradation, confirming the necessity of each component. Specifically, the turn-level advantage construction was shown to be critical, as broadcasting advantages (like standard GRPO) reduced performance to near outcome-only levels.
• Training Dynamics: SIOP reduced redundant search turns and response lengths while maintaining generation entropy at lower levels, indicating more efficient exploration.

Significance and Claims

The paper claims that SIOP successfully bridges the gap between outcome-level reliability and process-level credit assignment in settings where ground truth is unavailable. By treating clustered final answers as latent future states, the framework generalizes information-potential shaping from gold-answer supervision to verifier-free environments.

The authors emphasize that SIOP does not merely select better final answers but actively shapes the intermediate reasoning trajectory. The method allows agents to learn which intermediate interactions (e.g., specific search queries or tool calls) move the trajectory toward reliable future states, even when the correctness of those states is only inferred from the consistency and evidence support of the model's own samples. This offers a practical path for training interactive agents in open-ended domains where task-specific verifiers are infeasible to construct.