Technical Summary: Information-Aware Credit Assignment for Visually Grounded Long-Horizon Information-Seeking Agents

Problem Statement

Long-horizon reinforcement learning (RL) for information-seeking agents faces two primary bottlenecks: sparse credit assignment and unstable information representation.

1. Sparse Credit Assignment: In tasks requiring multi-step web browsing, terminal rewards only indicate whether the final answer is correct. They fail to identify which specific search or fetch actions acquired the critical information necessary for success. This makes it difficult to optimize intermediate retrieval decisions, as agents cannot distinguish between useful evidence acquisition and unproductive exploration.
2. Unstable Representation: Existing agents typically rely on text-based parsing pipelines (linearization, summarization, or chunking) to process webpages. These methods often discard layout structures, visual cues, and tables. Furthermore, the same webpage may be parsed, truncated, or summarized differently across trajectories due to varying context budgets or tool pipelines, rendering acquired evidence unidentifiable and incomparable across different agent rollouts.

Methodology

The authors propose Information-Aware Credit Assignment (ICA), a framework that treats long-horizon web agent learning as an evidence attribution problem. The methodology consists of three core components:

1. Visual Snapshot-Based Evidence Units

Instead of linearized text, the framework represents fetched webpages as rendered snapshots. These snapshots preserve the full visible content, including layout, tables, and multimodal cues.

• Stability: Snapshots serve as stable, content-level observations that allow for consistent identification and comparison of the same information source across different trajectories.
• Efficiency: By preserving layout, snapshots reduce token usage significantly (27%–65% less than parsed text) and mitigate context overflow issues common in long-horizon tasks.

2. Information-Aware Credit Assignment (ICA)

ICA is a post-hoc reward propagation method that assigns dense rewards to intermediate steps based on the utility of the information they introduced.

• Atomic Evidence Association: The system defines "atomic evidence units" (e.g., a specific search result or a full webpage snapshot). It calculates an association-based utility score ($\Delta_e$) for each unit by comparing the success rate of trajectories that acquired the unit versus those that did not.
  • $P^+_e$: Success probability given evidence $e$ is acquired.
  • $P^-_e$: Success probability given evidence $e$ is not acquired.
  • $\Delta_e = P^+_e - P^-_e$.
• Turn-Level Reward Propagation:
  • Tool Turns: If a search or fetch turn introduces a high-utility evidence unit (ranked in the top fraction of acquired units), it receives a positive binary reward.
  • Answer Turns: The final answer turn receives an asymmetric shaped reward based on correctness, penalizing failures less severely if the agent explored thoroughly (budget-aware shaping).
  • Advantage Calculation: Rewards are normalized and assigned as turn-level advantages, decoupling search quality supervision from final answer outcomes.

3. ICA-GSPO Optimization

The method integrates ICA with Group Sequence Policy Optimization (GSPO). Unlike standard GSPO which computes importance ratios over an entire trajectory, ICA-GSPO computes importance ratios independently for each turn. This design aligns with the multi-turn structure of agents and prevents large policy shifts in a single token from contaminating trajectory-level estimates. The final objective aggregates the surrogate losses of tool turns and the answer turn with a fixed weighting factor.

Key Contributions

1. Evidence Attribution Formulation: The paper reframes long-horizon web agent learning as an evidence attribution problem, optimizing retrieval actions based on the information they introduce rather than solely on terminal outcomes.
2. Stable Visual Observations: It introduces rendered webpage snapshots as a stable content-level observation modality, enabling consistent cross-trajectory comparison of fetched information.
3. ICA Mechanism: It proposes a post-hoc credit assignment method that estimates association-based utility scores for tool-acquired information and propagates dense rewards to the specific search and fetch turns that introduced high-utility evidence.
4. Performance Gains: The integration of ICA with GSPO demonstrates consistent improvements across multiple benchmarks, with an 8B model outperforming most reported open-source agents under 15B parameters.

Experimental Results

The authors evaluated the approach on four long-horizon information-seeking benchmarks: BrowseComp, GAIA, Xbench-DS, and Seal-0.

• Performance: The proposed Qwen3-VL-8B-ICA model achieved a Pass@1 accuracy of 25.0 on BrowseComp, 69.9 on GAIA, 71.0 on Xbench-DS, and 25.2 on Seal-0. This significantly outperformed text-based baselines (e.g., WebExplorer-8B scored 15.7 on BrowseComp) and was competitive with larger proprietary systems.
• Ablation Studies:
  • Snapshot vs. Text: Using snapshots in Supervised Fine-Tuning (SFT) yielded consistent improvements over text parsing, particularly on benchmarks sensitive to layout (GAIA, Xbench-DS).
  • ICA vs. GSPO: ICA consistently outperformed vanilla GSPO, with the most significant gains observed in long-horizon, high-noise tasks.
  • Evidence Sufficiency: Analysis showed that evidence selected by ICA (top-p% based on utility scores) achieved significantly higher answer accuracy (71.0%) compared to random or low-scoring baselines, validating the method's ability to identify answer-supporting evidence.
  • Token Efficiency: Snapshot-based approaches reduced context overflow rates to 0.0% across all datasets, whereas text-based baselines suffered from overflow rates up to 36.0% on BrowseComp.

Significance and Limitations

Significance:
The paper claims that grounding reinforcement learning in visually structured external observations and performing credit assignment at the granularity of acquired evidence is a critical step forward for training long-horizon web agents. It demonstrates that effective optimization requires not just rewarding correct answers, but explicitly assigning credit to the specific retrieval actions that introduced useful information.

Limitations:
The authors acknowledge two primary limitations:

1. Visual Redundancy: The framework currently relies on uncompressed visual representations fed directly into Vision Transformers, which introduces token redundancy and computational overhead, especially for visually dense or extremely long webpages.
2. Context Management: The system lacks sophisticated context management strategies (e.g., dynamic context reduction, memory-based sliding windows) to handle unbounded exploration, which can lead to context overflow and loss of early-retrieved evidence in very deep search tasks.

The authors suggest that future work should focus on visual token compression and adaptive memory management to address these efficiency bottlenecks.