🤖 AI

DragOn: A Benchmark and Dataset for Drag-Based GUI Interactions

The paper introduces DragOn, a comprehensive benchmark and dataset comprising 286K screenshots and 3.5M tasks across four domains to address the scarcity of drag-based GUI interaction data, demonstrating that fine-tuning on this dataset significantly enhances the performance of state-of-the-art vision-language models on complex computer-use tasks.

Original authors: Nathan Bout, Maxime Langevin, Ronan Riochet

Published 2026-06-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nathan Bout, Maxime Langevin, Ronan Riochet

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to use a computer. So far, we've been very good at teaching robots how to click a button. It's like teaching a child to point at a picture and say, "Touch that!" We have millions of examples of this, and the robots are getting pretty good at it.

But there is a much harder skill: dragging.

Think of dragging like picking up a heavy box and moving it to a new spot, or swiping your finger to scroll down a page, or grabbing the corner of a photo to make it bigger. These actions require the robot to understand not just where to touch, but how to move from point A to point B. Currently, robots are terrible at this. They are like toddlers who can point at a toy but can't figure out how to pick it up and carry it across the room.

The paper introduces a new tool called DragOn to fix this problem. Here is how it works, broken down simply:

1. The Problem: A Lack of "Drag" Practice

The authors noticed that while we have huge libraries of data for teaching robots to click, the data for teaching them to drag is tiny—about 10 to 100 times smaller. Because of this, even the smartest AI models (like the ones from OpenAI or Anthropic) struggle with tasks like highlighting text, resizing windows, or moving sliders. They get confused and often fail.

2. The Solution: "Rendering-as-Supervision"

To build a massive library of drag examples without hiring thousands of humans to click and drag manually, the authors invented a clever trick they call "Rendering-as-Supervision."

The Old Way: Imagine a human looking at a screen, drawing a box around a word, and writing down the coordinates. This is slow and expensive.
The DragOn Way: Imagine you have a magic blueprint of a document (like a PDF or a spreadsheet). The computer knows exactly where every letter and cell is because it built the document. Instead of looking at the picture and guessing, the computer just asks the blueprint: "Where is the word 'Hello'?" The blueprint says, "It's at these exact coordinates."

The computer then "renders" (draws) the image and automatically labels the start and end points of the drag action based on that blueprint. It's like having a GPS that knows the exact location of every street sign before you even drive down the road. This allowed them to create 3.5 million training tasks very quickly and cheaply.

3. The Dataset: Four New Playgrounds

They didn't just make one type of drag task; they built four different "playgrounds" for the robots to practice on:

Text Highlighting: Dragging over a sentence to select it (like highlighting a word in a textbook).
Cell Selection: Dragging over a grid in a spreadsheet to select a block of numbers.
Element Resizing: Grabbing the corner of a picture or window and pulling it to make it bigger or smaller.
Slider Manipulation: Grabbing a slider bar (like a volume control) and sliding it left or right.

In total, they created 286,000 screenshots for the robots to study.

4. The Results: Practice Makes Perfect

The authors tested this new dataset on the world's smartest AI models.

The Baseline: The best "off-the-shelf" AI models (like GPT and Claude) scored below 30% on these tasks. They were essentially failing more often than they were succeeding.
The Breakthrough: The authors took an open-source AI model and trained it specifically on their new DragOn dataset.
The Outcome: This trained model jumped to 35.3% accuracy. While that might not sound like a huge number, it was a massive improvement over the base model (which was only at 2.3%) and it actually beat the most expensive, proprietary models on three out of the four tasks.

The Bottom Line

The paper claims that by giving AI models a massive, high-quality library of "drag" examples—created using a smart, automated blueprint method—we can teach them to handle complex computer tasks much better. It proves that data is the key: even a standard open-source model can outperform the most expensive commercial models if it is trained on the right kind of practice data.

The authors hope this new benchmark and dataset will help future robots become reliable "computer users" that can do more than just click; they can finally learn to drag, drop, resize, and swipe just like a human does.

Technical Summary: DragOn: A Benchmark and Dataset for Drag-Based GUI Interactions

Problem Statement

While Vision-Language Models (VLMs) have achieved significant progress in "click-grounding" (mapping natural language instructions to single screen coordinates), they struggle with "drag-grounding." Drag-grounding requires outputting a pair of coordinates to execute continuous interactions such as drag-and-drop, swipes, highlighting, and resizing. Existing datasets for these interactions are an order of magnitude smaller than click-grounding corpora and are largely dominated by text highlighting tasks. Consequently, current state-of-the-art models perform poorly on complex, pervasive drag-based interactions found in real-world GUI workflows, such as spreadsheet cell selection, element resizing, and slider manipulation.

Methodology

Rendering-as-Supervision

The core methodological contribution is a principle termed "rendering-as-supervision." Instead of relying on noisy human annotation, OCR, or VLM-based labeling, the authors construct ground-truth data by exploiting the renderer's own geometry.

Mechanism: A deterministic renderer ( $R$ ) maps a structured source ( $S$ ) to an image ( $I$ ). A label map ( $\pi$ ) extracts pixel-space bounding boxes directly from the source structure or via controlled perturbations.
Advantages: This approach yields pixel-exact annotations at a fraction of the cost of traditional methods and supports arbitrary granularity (from characters to multi-paragraph spans) without the errors associated with OCR.
Implementation: Two types of label maps are used:
1. Analytic: Directly computes bounding boxes from source coordinates (e.g., PDF span coordinates, PPTX EMU units, HTML geometry).
2. Probe-Based: Runs the renderer twice; a perturbed source (e.g., a cell filled with a distinctive color) is rendered to visually locate the target, after which the original source is re-rendered for the final screenshot.

The DragOn Dataset

The authors instantiate this principle across four heterogeneous domains:

Text Highlighting: Derived from Wikipedia articles converted to styled DOCX and then PDF. Granularity ranges from characters to paragraphs.
Cell Selection: Synthetic spreadsheets generated from Pandas DataFrames, rendered via LibreOffice Calc. Includes range selections and border resizing.
Element Resizing: PowerPoint slides from the Zenodo 10K corpus. Geometry is extracted from XML (EMU units) and mapped to pixel space.
Slider Manipulation: Procedurally generated HTML pages with various interactive contexts (e.g., media players, settings panels).

Scale: The dataset comprises 286,000 training screenshots and 3.5 million training tasks, alongside a 2,000-example held-out evaluation suite (split into public validation and private test sets). This scale is one to two orders of magnitude larger than prior drag-specific corpora.

Key Contributions

Rendering-as-Supervision Principle: A formalized method for constructing GUI grounding data using renderer geometry as an oracle, reducing construction errors and costs.
DragOn Dataset: A unified, large-scale corpus covering four distinct drag domains, significantly expanding the available data for training and evaluating drag-grounding capabilities.
Benchmark and Baselines: A comprehensive evaluation of proprietary models (GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.7), open-weight models (Qwen, Kimi, Holo), and a model fine-tuned on the DragOn dataset.

Experimental Results

The authors evaluated models on a 2,000-example test set. Key findings include:

Frontier Model Performance: Proprietary models (GPT, Claude) and open-weight models generally score below 30% overall. Performance varies significantly by domain:
- Strongest: Slider manipulation (up to 57.2% for Claude Opus 4.7) and cell selection.
- Weakest: Text highlighting (often below 10%) and element resizing (often below 20%).
Open-Weight Baselines: Standard open-weight models (e.g., Qwen3.5-35B) perform poorly (2.3%), though larger variants (Qwen3.5-397B) approach proprietary levels (21.5%).
Impact of Fine-Tuning: A Qwen3.5-35B model fine-tuned on the DragOn dataset achieved a 35.3% overall success rate. This represents a 33-point absolute improvement over the base model and outperforms all evaluated frontier models by at least 7.6 points.
Domain Specificity: The fine-tuned model led in three of the four domains (text highlighting, slider manipulation, element resizing) but trailed slightly in cell selection compared to Claude Opus 4.7.
Tolerance Analysis: For element resizing, performance improves significantly under relaxed spatial tolerance (e.g., from 32.0% at strict thresholds to 57.2% at 15% tolerance), suggesting errors are often small-magnitude offsets rather than qualitative mis-groundings.

Significance and Claims

The paper claims that drag grounding remains a challenging, unsaturated problem for current VLMs, even for the strongest closed systems. The primary significance of the work lies in demonstrating that targeted training data can substantially improve model performance on downstream computer-use tasks.

The authors posit that the DragOn benchmark and dataset provide the necessary infrastructure to advance the development of reliable autonomous agents capable of performing complex, continuous GUI interactions. By showing that an open-source model fine-tuned on their data can surpass proprietary frontier models, the paper suggests that data quality and domain specificity are critical levers for improving agentic capabilities, potentially narrowing the gap between open and proprietary systems in computer-use scenarios.