💻 computer science

VISUALSKILL: Multimodal Skills for Computer-Use Agents

The paper introduces VISUALSKILL, a multimodal skill framework that enhances computer-use agents' performance on long-horizon tasks by retaining visual figures in skill artifacts rather than converting them to text, resulting in significant accuracy improvements over both baseline and text-only skill approaches.

Original authors: Ziyan Jiang, Li An, Yujian Liu, Jiabao Ji, Qiucheng Wu, Jacob Andreas, Yang Zhang, Shiyu Chang

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ziyan Jiang, Li An, Yujian Liu, Jiabao Ji, Qiucheng Wu, Jacob Andreas, Yang Zhang, Shiyu Chang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to use a complex piece of software, like a photo editor or a spreadsheet program. The robot can see the screen and move the mouse, but it doesn't "know" the software. It's like handing a tourist a map of a city they've never visited, but the map is written entirely in a foreign language and has no pictures. They might guess where to go, but they'll likely get lost, especially when the streets change or when they need to find a specific, hidden alleyway.

This is the problem the paper VISUALSKILL tries to solve.

The Problem: Text-Only Maps vs. Visual Reality

Current "computer-use agents" (robots that use software) are getting pretty good, but they still struggle with long, complicated tasks or software they haven't seen before.

To help them, researchers have built "skill libraries"—basically instruction manuals for the robot. But here's the catch: these manuals are written only in text.

The paper argues this is a bad idea for two reasons:

Describing pictures with words is hard: Imagine trying to tell a robot which icon to click just by saying "the blue brush tool." If there are five blue tools nearby, the robot might click the wrong one. A picture shows the exact shape and location instantly.
Checking your work is hard: In a multi-step task, the robot needs to check, "Did I actually open the right menu?" If the manual says, "You should see a small window pop up," the robot has to guess what that looks like. If the manual shows a screenshot of that exact window, the robot can just compare the screen to the picture and know for sure.

The Solution: VISUALSKILL

The authors created VISUALSKILL, a new way to build these instruction manuals. Instead of just text, VISUALSKILL keeps the original pictures and screenshots right next to the instructions.

Think of it like this:

Old Way (Text-Only): A recipe that says, "Add the red sauce." (The robot has to guess which bottle is red).
VISUALSKILL: A recipe that says, "Add the red sauce," and right below it, a photo of the exact red sauce bottle on the shelf.

How It's Built (The Two-Stage Pipeline)

The team didn't just write these manuals by hand; they built a system to create them automatically in two steps:

Stage 1: The Official Manual Miner.
They take the software's official user guide (the PDF or website the company made) and turn it into a structured skill. They keep all the original diagrams and screenshots from the guide. This gives the robot a solid foundation.
Stage 2: The Live Explorer.
Official guides are often outdated or miss the weird, small pop-up windows that only appear when you actually use the software. So, the system sends a "robot explorer" to actually use the software.
- Free Exploration: The explorer wanders around the idle screen, clicking buttons to see what happens, and takes photos of every new window that appears.
- Targeted Exploration: The system looks at tasks where the robot failed before. It figures out, "Ah, the robot got stuck on this specific menu," and sends the explorer specifically to that spot to take a picture and write a note about it.

These new photos and notes are then stitched back into the manual, making it a "living" guide that matches the real software perfectly.

How the Robot Uses It

The robot doesn't read the whole 100-page manual at once (that would be too much information). Instead, it has a special tool called load_topic.

The robot looks at its current task.
It checks a short index (like a Table of Contents) to see which topic is relevant.
It asks the tool: "Show me the instructions for this topic."
The tool instantly pulls up the text and the specific screenshots needed for that exact moment.

The Results

The researchers tested this on two major benchmarks (sets of difficult computer tasks) using a powerful AI agent.

No Skill: The robot had no help. It succeeded about 30% of the time.
Text-Only Skill: They gave the robot a text-only manual made from the same source. Success went up to 37%.
VISUALSKILL (Multimodal): They gave the robot the manual with pictures. Success jumped to 45%.

The Key Finding: The pictures made a huge difference. The robot was much better at:

Finding the right button: It could see the icon instead of guessing based on a description.
Checking its progress: It could compare the screen to the reference photo to see if it was on the right track.

Why This Matters

The paper concludes that for robots using computers, visuals are not just "nice to have"—they are essential. Trying to describe a graphical interface using only words is like trying to describe a painting to someone who has never seen one. By keeping the pictures in the instruction manual, the robot can actually "see" what it's supposed to do, just like a human would.

The authors also found that how the robot gets the information matters. If they just let the robot read files like a human reads a book, it often skipped the pictures. By using a special tool (load_topic) that delivers the text and pictures together in one go, the robot actually used the visual clues effectively.

In short: Don't just tell the robot what to do; show it.

Technical Summary: VISUALSKILL

Problem Statement

Computer-use agents (CUAs) have achieved near-human performance on standardized desktop benchmarks but continue to struggle with long-horizon tasks and unseen software. This limitation stems largely from a lack of persistent, application-specific procedural knowledge (e.g., specific menu locations, dialog sequences, and multi-step workflow states) that pre-training does not supply. While existing skill libraries attempt to address this by providing reusable knowledge, they predominantly represent skills as text-only artifacts. This modality is ill-suited for GUI interaction for two primary reasons:

Information Loss: Many UI elements (icons, layouts, spatial relations) are verbose or ambiguous to describe textually. Verbalizing these elements causes a loss of information present in the original screenshots.
Verification Difficulty: In multi-step workflows, agents must verify intermediate UI states after each action. Textual descriptions provide only indirect grounding, making it difficult to distinguish between the intended state and visually similar but incorrect states.

Methodology: VISUALSKILL

The authors propose VISUALSKILL, a hierarchical multimodal skill artifact designed to retain visual figures alongside textual procedures.

Skill Structure

A VISUALSKILL is organized as a two-layer hierarchy per target application:

Central Index (skill.md): Lists all topics with a short "when to use" description.
Per-Topic Guides: Each guide $g_t$ pairs a text body $p_t$ with a set of UI figures $F_t$ .
Loading Mechanism: The agent does not load the entire skill upfront. Instead, it consults the index and invokes a load_topic(t) Model Context Protocol (MCP) tool to fetch the specific text and figures for a topic on demand. This ensures figures are delivered inline with their referencing text, preventing them from being skipped or separated from context.

Construction Pipeline

VISUALSKILL is constructed via a two-stage pipeline that generates both the multimodal skill ( $S_{mm}$ ) and a matched text-only control ( $S_{txt}$ ) from the same source content to isolate the effect of modality:

Stage 1: Documentation Mining:
- Parses the application's official user guide (PDF/HTML).
- Reuses the document's table of contents as the topic hierarchy.
- Extracts vendor-drawn figures verbatim and generates text bodies for both modalities via a joint LLM call.
Stage 2: Live UI Exploration:
- Augments the Stage 1 skill by driving the live application with an LLM-controlled explorer to capture knowledge missing from documentation (e.g., low-frequency dialogs, visual affordances).
- Free Explorer: A planner partitions the idle application window into regions; worker agents interact with each region to capture screenshots and summarize affordances.
- Trajectory-Targeted Explorer: A reviewer analyzes failed training trajectories to identify UI regions where the Stage 1 skill underperformed. Workers are dispatched to these specific regions to capture missing dynamic states.
- Captured screenshots and notes are merged into the per-topic guides.

Key Contributions

Hierarchical Multimodal Artifacts: The design of VISUALSKILL as a topic-indexed structure where visual figures are retained alongside text, accessed via an on-demand MCP tool.
Two-Stage Construction Pipeline: A method combining authored documentation mining with live-application UI exploration (both free and trajectory-targeted) to create comprehensive skill sets.
Controlled Evaluation: The construction of a matched text-only control generated from the exact same source material and procedural steps, differing only in modality. This isolates the performance gain attributable specifically to the presence of visual figures.

Experimental Results

The authors evaluated VISUALSKILL using a Claude Code CLI agent backed by Claude Opus 4.6 across 177 tasks from CUA-World and OSExpert-Eval.

Baseline Improvement: The agent achieved an average score of 0.303 with no skills.
Stage 1 Impact: Using documentation-derived VISUALSKILL, the score rose to 0.363 (+6.0% absolute).
Stage 2 Impact: Adding UI exploration (Stage 2) further increased the score to 0.456 (+15.3% absolute over baseline).
Multimodal vs. Text-Only:
- At Stage 1, the multimodal skill outperformed the text-only control by +1.9% absolute (0.344 vs. 0.363).
- At Stage 2, the multimodal skill outperformed the text-only control by +8.3% absolute (0.373 vs. 0.456).
- The multimodal advantage was most pronounced in visually intensive domains (e.g., GIMP, OpenToonz, Tableau) where success depends on recognizing icons, dialogs, and layouts.
Ablation on Loading Mechanism: Replacing the load_topic MCP tool with a standard Read file access mechanism resulted in a significant drop in performance. The MCP tool ensured the agent consulted the skill throughout the trajectory (median last consult at step 10.4) and loaded ~7.9 figures per task, whereas the Read approach led to early abandonment (median step 1.5) and loading only ~0.8 figures per task.

Significance and Claims

The paper claims that retaining visual figures in skill artifacts, rather than verbalizing them, provides direct evidence that helps agents:

Identify UI elements: Precisely locating icons, layouts, and spatial relations that are ambiguous in text.
Verify workflow states: Directly matching the current observation against a reference screenshot to confirm intermediate states, particularly in silent failures (e.g., form fields that appear accepted but reject on commit).

The authors conclude that for computer-use agents, the skill artifact should be a multimodal, on-demand reference rather than a text-only document. They emphasize that the combination of visual grounding, the MCP-based delivery mechanism, and trajectory-targeted exploration are individually necessary to achieve the reported performance gains.

Limitations: The authors note that current models sometimes struggle to decide which specific figures within a loaded topic are relevant, leading to context inflation. In weaker models, this coarse granularity can degrade performance due to context-length pressure, suggesting that more effective context budgeting for multimodal skills remains an open question.