Technical Summary: RESOURCE2SKILL

Problem Statement

Large language model (LLM) agents are increasingly tasked with operating complex software to produce high-quality artifacts (e.g., slide decks, 3D scenes, audio projects). Success in these domains relies less on isolated factual knowledge and more on reusable procedural knowledge—skills that define how to decompose goals, select tool patterns, inspect intermediate states, and recover from failures.

Existing skill libraries suffer from three primary limitations:

1. Creation Bias: They are predominantly hand-written by experts or mined from agent interaction traces, leaving vast amounts of human expertise underutilized.
2. Modality Gap: Human learning for creative software tasks often occurs through multimodal resources (tutorial videos, screen recordings, repositories), which contain temporal sequences, visual effects, and tacit design choices that static text cannot fully capture.
3. Inference Inefficiency: Directly injecting raw multimodal data (e.g., full videos) into an agent's context is expensive and redundant, while compressing them into plain text summaries discards the dynamic and visual signals essential for execution.

The core challenge is to automatically distill these multimodal human-created resources into a structured, executable, and retrievable format that agents can effectively use at inference time.

Methodology

RESOURCE2SKILL is a framework designed to distill multimodal resources into a hierarchical Skill Wiki and utilize it for agent execution. The system operates through a unified pipeline involving construction, organization, selection, and execution.

1. Hierarchical Multimodal Skill Wiki

The system organizes skills as a structured tuple $s = (p, x_{text}, x_{visual}, x_{code}, m)$:

• $p$ (Path): A position in a domain-specific taxonomy (e.g., Blender skills organized by geometry, lighting, materials).
• $x_{text}$: Structured text explaining the mechanism, applicability, inputs, and expected effects.
• $x_{visual}$: Visual examples including thumbnails, screenshots, rendered previews, or diagrams.
• $x_{code}$: Executable or adaptable code fragments (or reference-only patterns).
• $m$ (Metadata): Filtering tags, provenance, and validation status.

This design ensures that text provides conceptual grounding, code offers executable patterns, and visuals preserve layout, style, and motion information that text alone under-specifies.

2. Resource-to-Skill Construction

A construction operator $(f_\theta, A_D)$ transforms raw resources into wiki entries:

• Input: Multimodal resources including tutorial videos, source repositories, articles, and reference artifacts.
• Distillation: A vision-capable language model ($f_\theta$) extracts modality-specific evidence (key frames, code regions, prose passages) and generates a structured skill payload.
• Validation: A domain-specific predicate ($A_D$) enforces five deterministic checks: completeness, traceable provenance, deduplication, modality consistency, and structural executability (for code fields).
• Online/Offline Unity: The same operator is used to build large offline libraries and to acquire new skills online when the offline pool lacks coverage, ensuring the library is incrementally extensible.

3. Selection and Composition (METABROWSE)

To retrieve skills efficiently, the system employs a two-stage selection process:

1. Lexical Scoring: A BM25 scorer narrows the candidate set by searching the entry's name, tags, applicability, and crucially, its taxonomy path ($p$). This leverages the hierarchical structure to favor skills in topically relevant subtrees rather than treating the library as a flat list.
2. LLM Selection: A language model reads the structured evidence (text, visual, code) of the top candidates and selects a subset to compose. The model can choose zero skills if no candidate fits, preventing forced composition.

4. Execution

Selected skills are applied via a Model Context Protocol (MCP) interface. If code is present, it executes directly against the domain backend. Reference-only skills guide the agent to write its own code based on the visual and textual evidence.

Key Contributions

1. Resource-to-Skill Learning: The paper formulates the problem of distilling reusable executable skills from multimodal human resources, identifying tutorial videos as a critical, underused source of procedural and perceptual knowledge.
2. Hierarchical Multimodal Skill Library: It proposes a Wiki-based organization where each skill integrates text, code, and visuals, enabling agents to retrieve and compose skills across diverse domains more effectively than flat text retrieval.
3. Unified Offline-Online Pipeline: The framework uses a single construction operator for both building large-scale offline libraries and performing controlled online gap-filling, making the skill library maintainable and extensible.
4. Broad Empirical Evaluation: The system is evaluated across seven practical authoring domains: PPT, Web, Excel, Blender, Reaper, CAD, and UE5.

Experimental Results

The evaluation covers seven domains and four agent backends (GPT-5.5, GPT-5.4, GPT-5.4 Mini, GPT-5.4 Nano), comparing W SKILLS (RESOURCE2SKILL) against W/O SKILLS (no-skill baseline) and two off-the-shelf agentic harnesses (CLAUDECODE-H, CODEX-H).

• Performance Gains: RESOURCE2SKILL improves the average overall score by +11.9 percentage points over agents without skills. It outperforms strong agentic harness baselines in 26 of 28 main-aggregate model–domain cells.
• Domain Variance: Gains are most pronounced in domains with dense conventions and complex visual/temporal requirements (e.g., UE5 saw +30 to +40 pp gains; Blender and Excel also showed significant improvements). Gains were smallest in Reaper, where the no-skill prior was already relatively competent.
• Ablation Studies:
  • Multimodal Format: Removing visual or code fields significantly degrades performance, confirming that text alone under-specifies execution.
  • Source Diversity: Video is identified as a non-substitutable source; removing it drops the average score from 68.9% to 59.4%.
  • Organization: The hierarchical Wiki interface outperforms flat text retrieval by 2.5–8.2 percentage points, particularly in domains requiring structural context.
  • Selection Strategy: The hierarchy-then-LM selection strategy (METABROWSE) outperforms pure lexical (BM25) or dense (Embedding) retrieval baselines.
  • Online Acquisition: Online acquisition acts as an effective gap-filler, improving scores by +21.6 pp on novel tasks missing from the offline pool, while adding negligible noise to standard tasks.

Significance and Claims

The paper claims that RESOURCE2SKILL demonstrates that distilling skills from human-created multimodal resources provides software agents with reusable procedural knowledge that significantly outperforms both no-skill agents and strong agentic harnesses.

Key takeaways emphasized by the authors include:

• Video is Critical: Tutorial videos provide temporal and visual signals that text cannot replace, making them essential for high-quality skill distillation in creative domains.
• Structure Matters: A hierarchical organization of skills is superior to flat retrieval, as it encodes domain structure and narrows the search space.
• Scalability: The system is not a fixed prompt collection but a growing, maintainable procedural memory that can be extended online when capability gaps are identified.
• Generalizability: The approach is validated across diverse software authoring tasks, suggesting a generalizable path for enhancing agent capabilities in complex, tool-rich environments.

The authors maintain modesty regarding generalization, noting that the system relies on the existence of programmatic tool interfaces and public streams of procedural content, and that judge-human agreement, while acceptable, is not perfect. The work focuses on the efficacy of the distillation and retrieval framework rather than claiming universal applicability to all agent tasks.