Technical Summary: Online Skill Learning for Web Agents via State-Grounded Dynamic Retrieval

Problem Statement

Language agents are increasingly utilized for multi-step web automation tasks (e.g., information seeking, form filling). While existing research has explored online skill learning—where agents induce reusable procedural knowledge from past task trajectories to improve future performance—a critical limitation persists in current methodologies.

Existing online skill learning approaches primarily operate at the task level. They retrieve a fixed set of skills based solely on the initial task instruction at the beginning of an execution and hold this set constant throughout the entire process. This static strategy is misaligned with the nature of interactive web execution, where the appropriate next action depends not only on the global task goal but also on the current webpage state. As the agent navigates, the webpage state transitions, often rendering initially selected skills irrelevant or failing to cover new situations that arise. Furthermore, existing methods struggle with skill granularity: full-trajectory skills are too specific to apply to intermediate states, while single-action skills lack meaningful procedural abstraction.

The core challenge addressed by this paper is: How can an online agent dynamically retrieve the right reusable skill at each decision step, conditioned on both the evolving task goal and the current webpage state?

Methodology: State-Grounded Dynamic Retrieval (SGDR)

The authors propose State-Grounded Dynamic Retrieval (SGDR), an online skill learning framework designed to replace static, task-level skill reuse with adaptive, step-level retrieval. The method operates within a sequential task-stream setting where agents must update their skill library on the fly without access to ground-truth success signals during execution.

SGDR consists of three core components:

1. Sliding-Window Skill Extraction

To address the granularity challenge, SGDR does not store entire task trajectories as single skills. Instead, after a task is completed and judged successful by an evaluator model, the system applies a sliding-window extraction process to the trajectory.

• Process: The trajectory is segmented into local sub-procedures (windows of varying lengths).
• Verification: Each candidate segment is converted into a skill and verified by replacing the corresponding actions in the original trajectory with a skill call. Only skills that preserve the trajectory's success are added to the library.
• Outcome: This produces compact, reusable sub-procedures (e.g., "fill a short form," "apply a filter") that are applicable to intermediate webpage states rather than just full-task contexts.

2. Dual Text–Code Representation

Each induced skill $s_k$ is represented as a pair $(d_k, c_k)$:

• Description ($d_k$): A natural language summary used for retrieval.
• Code ($c_k$): An executable function used for action execution.
  This dual representation ensures that skills are both semantically retrievable and functionally executable within the browser environment.

3. State-Grounded Dynamic Retrieval

Unlike static methods that retrieve skills once at the start, SGDR retrieves skills at every decision step $t$ based on the current state.

• Query Construction: The retrieval query combines the original task instruction $g$ and a compact summary of the current webpage state $r_t$ (derived from the accessibility tree).
• Relevance Scoring: A relevance score is computed using a weighted cosine similarity between the skill description and the combined query:
  $$score_t(s_k) = \alpha \cos(\phi(g), \phi(d_k)) + (1-\alpha) \cos(\phi(r_t), \phi(d_k))$$
  where $\alpha$ balances the importance of the task goal versus the current state.
• MMR Reranking: To prevent redundancy (since sliding windows may produce overlapping skills), the top-$M$ candidates are reranked using Maximal Marginal Relevance (MMR). This maximizes relevance to the current query while penalizing similarity to already selected skills, ensuring a diverse set of actionable options.
• Injection: The selected skills are injected only for the current step, allowing the agent's support to adapt as the execution unfolds.

Key Contributions

1. Problem Formulation: The paper identifies and formalizes the limitation of task-level one-shot skill reuse in online web agents, highlighting the need for state-grounded dynamic retrieval to handle evolving webpage contexts.
2. Methodological Innovation: It introduces SGDR, a framework that enables step-wise skill reuse. Key innovations include:
   • Sliding-window extraction to generate intermediate-granularity skills.
   • Dual text-code representation linking retrieval semantics to executable actions.
   • Dynamic retrieval mechanisms that condition skill selection on both task goals and real-time state observations.
3. Empirical Validation: The method is evaluated on the WebArena benchmark across five domains (Shopping, Admin, Reddit, Gitlab, Map) using two backbone models (GPT-4.1 and QWEN3-4B).

Experimental Results

Experiments demonstrate that SGDR consistently outperforms strong baselines, including Vanilla (no skills), Agent Workflow Memory (AWM), Agent Skill Induction (ASI), and Contextual Experience Replay (CER).

• Success Rates:
  • With GPT-4.1, SGDR achieved an average success rate of 37.5%, a 10.6% relative gain over the strongest baseline (CER at 33.9%).
  • With QWEN3-4B, SGDR achieved 24.3%, a 10.0% relative gain over CER (22.1%).
  • Improvements were observed across four of the five domains, with notable gains in Admin (47.7% vs. 41.4% for ASI) and Shopping.
• Execution Efficiency: SGDR reduced the average number of steps required to complete tasks. With GPT-4.1, it used 4.8 steps on average compared to 6.4 for CER, indicating that high-level skill abstraction effectively replaces multiple low-level browser interactions.
• Ablation Studies:
  • Retrieval Signal: Combining task and state information ($\alpha=0.5$) yielded the best results, outperforming task-only or state-only retrieval.
  • Reranking: MMR reranking significantly improved performance over simple top-$M$ relevance retrieval by reducing redundancy.
  • Granularity: Sliding-window extraction outperformed both full-trajectory and single-action extraction, confirming the necessity of intermediate granularity.

Significance and Claims

The paper claims that SGDR represents a shift from static task-level reuse to adaptive state-grounded reuse. By dynamically aligning skill retrieval with the evolving webpage state, the method allows agents to better exploit accumulated procedural knowledge in real-world, interactive environments.

The authors position this work as a step toward more realistic web automation, where agents must handle tasks that arrive sequentially and require continuous adaptation. The method does not rely on pre-constructed offline libraries but rather learns and refines its capabilities "on the fly," making it suitable for deployment scenarios where task distributions may shift or evolve. The results suggest that addressing the mismatch between static skill injection and dynamic web states is crucial for improving the success rates and efficiency of language agents in complex web environments.