💬 NLP

MMG2Skill: Can Agents Distill In-the-Wild Guides into Self-Evolving Skills?

This paper introduces MMG2Skill, a closed-loop framework and corresponding benchmark (MMG2Skill-Bench) that enables AI agents to distill noisy, multimodal web guides into self-evolving executable skills through structured compilation and trajectory-driven revision, significantly outperforming baseline agents across diverse domains.

Original authors: Xinyu Che, Junqi Xiong, Yunfei Ge, Xinping Lei, Shihao Li, Hang Yan, Han Li, Yuanxing Zhang, Zhiqi Bai, Jinhua Hao, Ming Sun, Han Li, Jiaheng Liu

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Xinyu Che, Junqi Xiong, Yunfei Ge, Xinping Lei, Shihao Li, Hang Yan, Han Li, Yuanxing Zhang, Zhiqi Bai, Jinhua Hao, Ming Sun, Han Li, Jiaheng Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant but slightly naive robot assistant. You want it to perform complex tasks like editing a photo, building a house in a video game, or playing a strategic card game. You have a massive library of human-written guides, tutorials, and wikis on the internet that explain how to do these things.

The problem is that these guides are written for humans, not robots. They are messy, assume you know things the robot doesn't, and often get confused if the robot makes a small mistake. If you just hand the robot a raw link to a "How to Build a House" article, it might get overwhelmed by the text, miss the crucial steps, or try to do things in the wrong order.

This paper introduces MMG2Skill, a new way to teach robots using these messy human guides. Think of it as a "Translator and Coach" system that turns human instructions into a robot's personal playbook.

Here is how it works, broken down into three simple stages:

1. The Translator: Turning "Human Talk" into "Robot Playbook"

Instead of giving the robot the whole messy article, the system first reads the guide and extracts the core steps. It turns the article into a clean, structured checklist called a Skill.

The Analogy: Imagine a human chef reading a complex recipe blog post. Instead of handing the whole blog to the sous-chef, the head chef writes a concise, bulleted "SOP" (Standard Operating Procedure) card: "Step 1: Chop onions. Step 2: Heat pan. Step 3: Add oil."
The Result: The robot now has a clear, editable instruction card (a SKILL.md file) rather than a confusing wall of text.

2. The Coach: Learning from Mistakes in Real-Time

The robot tries to do the task using this new checklist. If it fails, the system doesn't just say "Try again." It acts like a detective.

The Detective: It looks at exactly what the robot did, compares it to the goal, and figures out why it failed. Did the robot forget to wait for the file to save? Did it click the wrong button?
The Fix: The system then edits the checklist. It adds a note like, "Wait 5 seconds for the file to save before clicking 'Done'."
The Analogy: It's like a driving instructor watching a student fail a parallel parking test. Instead of just saying "You failed," the instructor writes a note on the student's practice sheet: "Remember to check the rearview mirror before reversing." The next time, the student uses that updated sheet.

3. The Smart Stopper: Knowing When to Quit

Sometimes, a robot keeps trying even after it has succeeded, or it keeps failing in a loop. The system includes a "Smart Stopper."

The Analogy: Imagine a GPS that realizes you've already arrived at your destination. Instead of making you drive around the block five more times just to be sure, it says, "You're here. Stop the engine."
The Benefit: This saves time and money (computing power) by stopping the robot as soon as it sees clear signs of success, rather than forcing it to run a fixed number of attempts.

What Did They Find?

The researchers tested this on three very different worlds:

Desktop Computers: Controlling software like Word or Photoshop.
Video Games: Playing Minecraft to gather resources and craft items.
Card Games: Playing strategy games like Doudizhu or Mahjong.

The Results:

Raw Guides Hurt: When they just gave the robot the raw, messy human guides, the robot actually got worse at the tasks. The extra noise confused it.
The Playbook Wins: When they used the "Translator and Coach" system (MMG2Skill) to turn those guides into clean, editable skills, the robots got significantly better. They improved by 12% to 25% across the board.
The Magic of Editing: The biggest gains came from the system's ability to fix the checklist after a mistake. A one-time translation wasn't enough; the robot needed to learn from its own failures and update its own playbook.

The Bottom Line

This paper shows that we don't need to write perfect code for every robot task. We can use the millions of human guides already on the internet, but we must translate them into a format the robot understands and let the robot edit them as it learns. It turns a messy human manual into a self-improving robot playbook.

Technical Summary: MMG2Skill

Problem Statement

Vision-language model (VLM) agents are increasingly tasked with long-horizon procedural work in interactive environments, such as desktop GUI control, open-world gaming, and strategic card play. A primary obstacle to success in these domains is weak procedural grounding: the inability of agents to anchor reusable procedures in their runtime state. While the web contains abundant multimodal procedural knowledge (guides, tutorials, documentation), this material is designed for human executors. It is often multimodal, heterogeneous, noisy, and implicitly assumes a human context (e.g., sibling procedures, prerequisite assumptions, navigation text).

Directly prompting agents with raw guides often fails because the agent cannot easily distinguish applicable steps from irrelevant content or adapt the guide when the agent deviates from the nominal path. Conversely, hand-writing skills for every task is labor-intensive, and skills discovered solely through agent exploration are limited by the agent's own failures. The central question addressed by this work is: Can VLM agents distill in-the-wild multimodal guides into execution-grounded skills that improve through their own rollouts?

Methodology: MMG2Skill Framework

The authors propose MMG2Skill, a closed-loop framework designed to convert multimodal guides into editable, executable skills and continuously improve them without relying on benchmark scores during the revision process. The framework operates in three stages:

Multimodal Skill Construction (Tutorial-to-Skill):
The system normalizes raw multimodal guides (HTML, images, screenshots) into an editable SKILL.md representation. A VLM skill extractor converts the guide into a set of structured skills $S = \{z_i\}$ , where each skill $z_i$ contains:
- $u_i$ : A reusable procedure.
- $c_i$ : Applicability conditions.
- $v_i$ : Expected-state cues.
- $q_i$ : Recovery knowledge.
  This step transforms unstructured human artifacts into a safer procedural prior.
Skill-Conditioned Execution:
During an agent rollout (attempt $k$ ), the current skill set $S_k$ is injected into the agent's context alongside the task instruction and recent history. The agent's policy $\pi_\theta$ remains fixed; it conditions its actions on the task, history, and the current skills: $a_t \sim \pi_\theta(a_t | h_t, I, S_k)$ . The skill set acts as procedural context without altering the domain-specific action interface.
Trajectory-Driven Skill Revision:
After an attempt, an Analyzer reads only the task instruction and the agent-visible trajectory (no benchmark scores or hidden states). It produces a diagnosis $\rho_k = (e_k, r_k)$ , where $e_k$ is evidence of what worked or failed, and $r_k$ is an outcome assessment (e.g., likely_success).
A Refiner then uses the original guide, the current skills, and the accumulated diagnoses to update the skill set: $S_{k+1} = \text{REFINE}(G, I, S_k, \rho_{1:k})$ . The refiner edits the SKILL.md files by adding missing checks, sharpening state cues, or removing misleading advice, preserving validated behaviors from previous attempts.

The framework includes an early-stopping mechanism based on the analyzer's likely_success signal, which halts the loop when the trajectory appears to satisfy the task, preventing late-stage performance regressions.

Benchmark: MMG2Skill-Bench

To evaluate this capability, the authors introduce MMG2Skill-Bench, the first benchmark specifically designed for guide-to-skill learning.

Scope: It spans three interaction regimes: GUI (OSWorld), Game (OpenHA Minecraft), and Strategy (Doudizhu and Mahjong via RLCard).
Composition: It contains 130 success-inferable tasks paired with task-relevant public multimodal guides.
Constraints: Tasks are selected such that outcomes can be inferred from the agent-visible trajectory or public final state. The benchmark excludes tasks where answers can be directly copied from guides, ensuring the evaluation measures procedural grounding rather than retrieval.

Key Results

The framework was evaluated across six VLM backbones (including Claude-Opus, GPT-5.5, Kimi, Gemini, and Qwen) on MMG2Skill-Bench.

Overall Performance: MMG2Skill consistently outperformed vanilla baseline agents (no skills) and "Raw Guide" baselines (direct guide injection) in every model–domain setting. Macro-average gains ranged from +12.8 to +25.3 percentage points across backbones.
Mechanism Ablation:
- Raw Guide Injection: Directly prompting with raw guides was unreliable and often degraded performance (e.g., -1.67 pp on Game) due to guide-environment grounding mismatches.
- Structured Extraction: Converting guides into structured skills (without revision) provided a safer procedural prior, improving performance over vanilla agents.
- Trajectory-Driven Revision: The revision component contributed the majority of gains in Game and Strategy domains (over 90% of total gain), repairing gaps between the guide and runtime execution.
Efficiency: MMG2Skill improved scores while reducing the average number of displayed-attempt steps, indicating fewer exploratory detours.
Early Stopping: Analyzer-based early stopping prevented late-stage performance regressions (non-monotonicity) and saved 25%–53% of attempts on success-inferable tasks while maintaining high precision (>74%) and recall (>95% on GUI/Game).

Significance and Claims

The paper claims two primary contributions:

MMG2Skill-Bench: A novel benchmark pairing success-inferable interactive tasks with in-the-wild multimodal guides, establishing a standard for evaluating guide-to-skill learning.
MMG2Skill Framework: A closed-loop system that demonstrates public guides can be converted into editable, execution-grounded skills that improve through agent rollouts without requiring benchmark scores for revision.

The authors emphasize that access to guides alone is insufficient. Raw guide prompting can hurt performance, whereas structured skill construction provides a safer prior, and trajectory-driven revision is necessary to repair grounding gaps. Furthermore, they highlight that revision gains are non-monotonic, making calibrated early stopping essential for deployment. The work isolates the problem of converting procedural material into executable skills, distinct from the upstream problem of guide discovery.