🤖 AI

MMSkills: Towards Multimodal Skills for General Visual Agents

The paper introduces MMSkills, a framework that addresses the limitations of text-based skill representations by formalizing and implementing reusable multimodal procedural knowledge—combining textual procedures with state-conditioned visual evidence—to enhance the runtime decision-making capabilities of general visual agents.

Original authors: Kangning Zhang, Shuai Shao, Qingyao Li, Jianghao Lin, Lingyue Fu, Shijian Wang, Wenxiang Jiao, Yuan Lu, Weiwen Liu, Weinan Zhang, Yong Yu

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kangning Zhang, Shuai Shao, Qingyao Li, Jianghao Lin, Lingyue Fu, Shijian Wang, Wenxiang Jiao, Yuan Lu, Weiwen Liu, Weinan Zhang, Yong Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Teaching Robots to "See" Like Humans

Imagine you are teaching a robot to use a computer.

The Old Way (Text-Only Skills): You give the robot a written recipe: "Click the 'File' menu, then 'New', then 'Sheet'."
- The Problem: If the robot is on the wrong screen, or if a pop-up window is blocking the menu, the robot blindly follows the text. It clicks "File" even though the menu isn't there, gets stuck, and fails. It knows what to do, but not when to do it or what it looks like when it's done.
The New Way (MMSkills): You give the robot a "Smart Guide." This guide doesn't just have text; it has photos of what the screen should look like at every step, along with checklists to verify if the robot is on the right track.

This paper introduces MMSkills, a system that turns these "Smart Guides" into reusable tools for AI agents (robots that use computers or play games).

The Three Main Problems They Solved

The authors identified three big hurdles in making these Smart Guides work:

What goes inside the package?
- Analogy: If you are teaching someone to bake a cake, a text recipe isn't enough. You need a photo of the batter (to know when it's ready), a photo of the oven (to know it's hot), and a checklist (to ensure you didn't forget the eggs).
- The Solution: An MMSkill package contains three things:
  - The Text: The step-by-step instructions.
  - State Cards: A "traffic light" system. It tells the agent: "Go ahead only if you see a blue button," or "Stop! Do not click if you see a red error message."
  - Visual Evidence: Photos (keyframes) showing exactly what the screen should look like at critical moments (e.g., "Before" and "After" shots).
Where do we get these guides?
- Analogy: You don't want to hire a human to write a new recipe for every single cake. You want to watch people bake cakes, figure out the common steps, and write a generic recipe yourself.
- The Solution: They built an automated "Generator." It watches thousands of public videos of people using computers (trajectories), groups similar tasks together, and automatically writes these Smart Guides. It doesn't just copy the video; it extracts the logic and the visual cues.
How does the robot use them without getting confused?
- Analogy: Imagine trying to read a huge, 50-page photo album while driving a car. It's distracting and dangerous. The robot might get so focused on the old photos that it crashes into the real world.
- The Solution: They invented "Branch Loading."
  - Instead of shoving the whole photo album into the robot's main brain, the robot opens a temporary side window (a branch).
  - In this side window, it quickly looks at only the specific photo it needs right now to make a decision.
  - It then closes the side window and tells the main robot: "Okay, I checked the photo. You are on the right track. Now, click the button."
  - This keeps the main robot focused on the live screen, not the old reference photos.

How It Works in Real Life (The "Chart" Example)

The paper uses a specific example to show why this is better: Creating a Chart in a Spreadsheet.

Scenario: The task is "Create a chart on Sheet 2."
Text-Only Agent: Reads "Create chart." It sees a chart being made. It clicks "Done."
- Result: It created the chart on Sheet 1 (the wrong sheet) because the text didn't tell it to check which sheet was active. Score: 0.
MMSkills Agent:
1. It loads the "Create Chart" skill.
2. The State Card says: "Check: Is the active sheet named 'Sheet 2'?"
3. The Visual Evidence shows a photo of the correct sheet tab.
4. The robot sees it is currently on Sheet 1. The "Branch" says: "Wait! You are on the wrong sheet. Switch to Sheet 2 first."
5. The robot switches, creates the chart, and verifies the title matches the photo.
- Result: Perfect chart on the right sheet. Score: 1.

What the Results Showed

The researchers tested this on two types of tasks:

Desktop Tasks: Like using Excel, Chrome, or Word (OSWorld, macOSWorld).
Game Tasks: Like playing Minecraft or Super Mario Bros.

The Findings:

Better Success Rates: Robots using MMSkills solved significantly more tasks than robots with no skills or just text skills.
Helped Smaller Robots: Even smaller, less powerful AI models got much better at tasks when given these visual guides.
Fewer Mistakes: The robots made fewer repetitive errors (like clicking the same button 10 times) and finished tasks faster.
Not Just for Computers: The system worked just as well in video games, proving that "seeing the state" is important whether you are managing files or jumping over obstacles.

Summary

MMSkills is a way to teach AI agents not just what to do, but how to recognize if they are doing it right. By combining text instructions with "traffic light" checklists and specific photos, and by using a "side-window" method to look at them, the system helps robots avoid getting lost, confused, or stuck on the wrong screen. It turns a robot that blindly follows orders into a robot that actually understands the visual world it is working in.

Technical Summary: MMSkills: Towards Multimodal Skills for General Visual Agents

1. Problem Statement

Existing agent skill frameworks primarily encode reusable behaviors as textual prompts, executable code, or learned routines. While effective for agents operating in language-abstraction environments, this approach is insufficient for visual agents (e.g., GUI or game agents) whose decisions rely heavily on visual evidence. For these agents, procedural knowledge is inherently multimodal: successful reuse depends not only on what operation to perform but also on recognizing the relevant visual state, interpreting evidence of progress or failure, and deciding the next step based on visual cues.

Current text-only skills often become verbose yet underspecified for visual tasks, while raw demonstrations are lengthy, instance-specific, and difficult to adapt. This gap creates three specific challenges:

Representation: How to structure a multimodal skill package that binds procedures, visual states, and verification cues into a coherent, reusable unit.
Generation: How to derive such packages from public, non-evaluation interaction trajectories without relying on hand-written examples or raw replay.
Utilization: How to consult multimodal skill evidence at inference time without overwhelming the agent with excessive image context or causing the agent to over-anchor to reference screenshots rather than the live environment.

2. Methodology

The authors propose MMSkills, a framework comprising three core components: a Multimodal Skill Package, a Skill Generator, and a Branch-Loaded Skill Agent.

2.1 Multimodal Skill Package

An MMSkill is defined as a state-conditioned procedure package $M = (D, P, S, K)$ :

$D$ (Descriptor): A compact description of the skill.
$P$ (Procedure): A reusable textual description of the action pattern.
$S$ (State Cards): A set of runtime state cards $S_j$ $S_{j}$ that act as decision nodes. Each card specifies:
- When-to-use and When-not-to-use conditions.
- Visible cues to inspect.
- Verification cues to confirm progress or completion.
- Available views (e.g., full frame, focus crop, before/after).
$K$ (Keyframes): Multi-view image bundles aligned with state cards, including full frames for context, focus crops for specific cues, and optional before/after views for transitions.

This structure transforms a skill from a static instruction into a dynamic, state-aware guide where visual evidence helps the agent decide whether to follow, skip, or verify a procedure.

2.2 Skill Generation Pipeline

The framework introduces an automated Trajectory-to-Skill Generator that converts public, non-evaluation trajectories into MMSkills. The process involves five stages:

Embedding & Clustering: Grouping broad domains into semantically focused clusters.
Cluster-Level Planning: An LLM-based agent proposes atomic skills with workflow boundaries and completion conditions.
Merging: Deduplicating and generalizing plans into merged skill specifications.
Text-First Drafting: Generating descriptors, procedures, and state cards without reading images.
Image Grounding & Audit: Reading selected keyframes to ground focus regions, construct multi-view bundles, and audit the final packages using meta-skills (reusable scripts and quality gates).

2.3 Branch-Loaded Agent

To address the issue of context overload and visual anchoring, the authors propose branch loading. Instead of injecting the entire skill package (text + multiple images) into the main agent's context, the system operates in two stages within a temporary branch:

Stage 1 (Gated View Selection): The branch analyzes the live observation and the skill's text/state cards to determine if visual evidence is needed. If so, it selects only the relevant state cards and specific view types (e.g., focus crop vs. full frame).
Stage 2 (Planner Guidance): The branch aligns the selected evidence with the live state and returns a compact, structured guidance tuple $G_t$ (containing applicability, subgoals, plans, constraints, and verification checks) to the main agent.
The main agent uses this distilled guidance to make grounded decisions based on the live screenshot, preventing it from being distracted by reference images.

3. Key Contributions

Multimodal Skill Package: The first formulation of reusable skills for general visual agents as multimodal procedural knowledge, integrating textual procedures, runtime state cards, and multi-view keyframes.
Agentic Generator: An automated pipeline that transforms public interaction trajectories into multimodal skill packages through workflow grouping, procedure induction, visual grounding, and meta-skill-guided auditing.
Branch Loading: A runtime mechanism that isolates skill-environment grounding in a temporary branch, selecting and aligning evidence before returning structured decision support to the main agent.
Empirical Validation: Demonstration of significant performance gains across diverse GUI and game-based benchmarks (OSWorld, macOSWorld, VAB-Minecraft, Super Mario Bros) and multiple model families.

4. Experimental Results

The authors evaluated MMSkills on OSWorld, macOSWorld, VAB-Minecraft, and Super Mario Bros using frontier models (e.g., Gemini 3.1 Pro, Qwen3-VL-235B) and smaller models (e.g., Qwen3-VL-8B).

Performance Gains: MMSkills consistently improved success rates over both "no-skill" and "text-only skill" baselines. For example, on OSWorld, MMSkills improved Qwen3-VL-235B from 21.34% to 39.17% and Qwen3-VL-8B from 10.78% to 25.40%.
Ablation Studies:
- Components: Both state cards and multi-view visual evidence were shown to be critical; removing either degraded performance.
- Loading Mechanism: Branch loading outperformed direct loading (inserting full skills into context). Direct loading often hurt performance due to context pollution and visual anchoring, whereas branch loading provided clear gains even for text-only skills.
Interaction Dynamics:
- Invocation: MMSkills were invoked more frequently than text-only skills, suggesting state cards make relevance easier to detect.
- Efficiency: MMSkills reduced the average number of interaction steps (e.g., -5.35 steps for Qwen3-VL-235B on OSWorld), indicating shorter task-solving paths.
- Behavioral Shift: Agents using MMSkills exhibited fewer repetitive actions and a shift from exploratory trial-and-error to grounded, state-aware execution. For instance, the share of "click" actions decreased while "keyboard" and "DONE" actions increased, suggesting better completion awareness.

5. Significance and Claims

The paper claims that external multimodal procedural knowledge complements model-internal priors. By explicitly binding action procedures to the visual evidence and state-dependent decisions required for their application, MMSkills enable visual agents to:

Recognize when a procedure is applicable based on visual state.
Distinguish between progress, failure, and completion using visual cues.
Avoid the pitfalls of text-only skills (underspecification) and raw demonstrations (lack of adaptability).

The authors emphasize that this approach is particularly valuable for weaker visual agents, where explicit visual procedural knowledge can compensate for limited internal priors. The work suggests that for general visual agents, the "skill" unit must evolve from text/code to a state-conditioned multimodal package that guides decision-making through visual grounding.

Limitations: The authors acknowledge dependence on source-trajectory coverage, potential errors in skill generation or visual grounding, and the extra inference cost introduced by the branch-loading mechanism. They note that extending this to safety-critical settings would require stronger verification and online repair mechanisms.