Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining

Technical Summary: Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining

Problem Statement
Computer-Using Agents (CUAs) operate on Graphical User Interfaces (GUIs) through primitive actions (clicking, typing, scrolling). As these agents transition from single-step tasks to complex workflows, repeated patterns emerge that can be packaged as "skills"—named procedures sitting above primitive actions but below full task plans. While explicit skill libraries (referred to here as SKILL.md files) facilitate inspection and debugging, they are typically hand-written, creating a bottleneck for naming, scoping, and updating as interfaces evolve.

The central question of this work is whether explicit skill libraries can be automatically mined from interaction trajectories to improve downstream agent policies. The authors distinguish between the ability to discover coherent clusters of actions and the ability to transfer those clusters into a useful policy for new tasks. They note that a cluster can be coherent without being useful, and policy improvements may stem from increased data volume rather than the acquisition of a transferable skill vocabulary.

Methodology
The paper proposes a three-stage pipeline to mine skills from the InteraSkill (IW) dataset and evaluate their utility:

1. Phase 1: Trajectory Segmentation (Boundary Detection)

   • The pipeline segments trajectories based on "action jumps." It calculates the Euclidean distance between adjacent action vectors ($\Delta a_t = \|a_t - a_{t-1}\|_2$).
   • A threshold $\theta$ is selected on held-out IW data to maximize boundary F1. If $\Delta a_t > \theta$, a boundary is marked.
   • Action vectors consist of 15 normalized features (primitive action one-hot, screen coordinates, timestamp, text length, scroll amount) but exclude DOM or screenshot data.

2. Phase 2: Skill Embedding (Library Construction)

   • Segment Representation: Each segment is summarized as a "bag-of-actions" using the mean ($\mu$) and diagonal variance ($\Sigma$) of its action vectors. This representation is length-invariant but discards sequential order (e.g., click-before-type vs. type-before-click).
   • Clustering: Segments are clustered using the squared Bures distance (a special case of the 2-Wasserstein distance for diagonal Gaussians) between their mean/variance profiles. Agglomerative clustering is performed with $k=8$ to $16$.
   • Refinement: A supervised-contrastive encoder (MLP) is trained using the cluster assignments as pseudo-labels. This refines the embeddings to be more internally consistent, mapping 30-dimensional inputs (mean + variance) to a 16-dimensional latent space.

3. Phase 3: Skill-Aware Policy Training

   • The mined skill labels are used to train a policy for skill-sequence composition (predicting the next skill label given a task context).
   • Baselines: Simple MLP and Transformer models consume the learned segment embeddings.
   • Main Policy: Qwen3-8B is trained using Group Relative Policy Optimization (GRPO) from a base model. The reward model scores full skill-aware responses, and the policy is updated without supervised warm starts.
   • Evaluation: The primary metric is skill-step accuracy and exact sequence match on held-out benchmarks (WebArena, BrowseComp+), with IW serving as the source domain.

Key Contributions

1. Pipeline for Automated Skill Mining: The authors present a reproducible pipeline that segments GUI trajectories, clusters them into candidate skills, and generates SKILL.md-style artifacts.
2. Diagnostic Evaluation of Transfer: The study rigorously evaluates whether mined skills improve downstream policy performance, comparing learned models against zero-shot baselines and trivial statistical priors.
3. Negative Results and Limitations: The paper reports that while the mined structure is readable in the source domain, the current pipeline fails to produce reliable cross-domain policy improvements.

Results

• Source-Domain Readability: On the source IW benchmark, the mined clusters are highly readable. Five of eight clusters achieve $\ge 0.95$ purity against ground-truth IW skill labels (e.g., document_edit, data_transfer). The contrastive encoder improves the Normalized Mutual Information (NMI) from 0.65 (Wasserstein baseline) to 0.86.
• Poor Transfer Performance:
  • GRPO Performance: Training Qwen3-8B with GRPO on mined skills yields only marginal gains on the source domain (IW skill-step accuracy increases from 18.5% to 20.5%) and degrades or remains unchanged on held-out domains (WebArena drops from 55.8% to 44.2%; BrowseComp+ remains at ~43.3%).
  • Baseline Failure: A trivial Frequency baseline (predicting the most common next skill) outperforms both the learned MLP and the GRPO-trained policy on IW skill-step accuracy (34.9% vs. 23.3% and 20.5%, respectively).
  • Edit Distance: The auto-generated SKILL.md files have higher normalized edit distances than the Frequency baseline across all data sizes, failing to beat the simplest statistical prior.
• Boundary Detection Issues: The action-jump heuristic over-splits skills (low precision) and is not domain-stable. Applying the IW-derived threshold to WebArena results in a sharp drop in F1 score (0.119), indicating the boundary signal does not transfer.
• Representation Limitations: The "bag-of-actions" representation (mean/variance) discards sequential order, which is critical for GUI skills (e.g., copy before paste). This likely contributes to the inability of the policy to learn complex composition rules.

Significance and Claims
The authors frame this work explicitly as a diagnostic study rather than a success claim. The significance lies in clarifying the limits of current trajectory-mining approaches:

• Inspectability vs. Utility: Trajectory mining can expose inspectable skill structures in the source domain, but readability does not imply transferability.
• Pipeline Insufficiency: The current combination of an $\ell_2$ boundary detector, orderless segment representation, and offline reward model is insufficient for reliable cross-domain policy improvement.
• Need for Stronger Controls: The paper argues that future skill discovery work must compare against frequency and transition priors to ensure improvements are not merely artifacts of dataset imbalance or output-format adaptation.
• No Cross-Domain Gains: The study finds no evidence that the current learned components outperform trivial baselines or manual design in a cross-domain setting.

The paper concludes that while mined skills serve as useful inspectable structures and training signals for reward models, they are not yet a reliable replacement for statistical baselines or manual design for building robust computer-using agents.