WARC-Bench: Web Archive Based Benchmark for GUI Subtask Executions

The paper introduces WARC-Bench, a novel benchmark using Web ARChive files to evaluate multimodal AI agents on complex GUI subtasks, demonstrating that while current frontier models struggle, open-source models significantly improve through supervised fine-tuning and reinforcement learning with verifiable rewards to achieve competitive performance.

The Gist

Imagine you are teaching a robot how to use a computer. Most previous tests asked the robot to do one of two things: either point at a single button on a screen ("Click the red button") or plan a massive, complex journey ("Book a vacation for a family of four, including flights, hotels, and car rentals, all under $2,000").

The authors of this paper realized there was a huge gap in the middle. They noticed that before a robot can book that vacation, it has to master the tiny, tricky steps in between: scrolling through a list to find a specific date, dragging a slider to adjust a budget, or filling out a form without accidentally deleting the text already there. They call these "GUI subtasks."

Here is a simple breakdown of their work, WARC-Bench:

1. The Problem: The "Missing Middle"

Think of a complex web task like baking a cake.

• Visual Grounding: "Pick up the egg." (Too simple).
• Long-Horizon Navigation: "Bake a cake, frost it, and deliver it to a party." (Too complex, too many variables).
• The Missing Middle: "Crack the egg into the bowl without getting shell in it," or "Whisk the batter until it's smooth."

The authors argue that current AI robots are failing at these "middle steps." They might know what a cake is, but they struggle with the specific, fiddly mechanics of the kitchen tools.

2. The Solution: A "Time-Traveling" Test Kitchen

To test these robots, the team built WARC-Bench.

Usually, testing robots on the real internet is chaotic. Websites change, pop-ups appear, and servers crash. To fix this, the team used WARC files (Web Archives).

• The Analogy: Imagine taking a perfect, frozen snapshot of a website at a specific moment in time, including all its buttons, scripts, and images. You put this snapshot in a "time capsule."
• How it works: When they test a robot, they don't send it to the live internet. They send it into this "time capsule." The robot interacts with this frozen, perfect copy of the website. It's like a flight simulator for web browsers: safe, repeatable, and exactly the same every time.

They created 438 different "mini-challenges" in this simulator, like "Select March 21st on the calendar" or "Scroll down to find the price."

3. The Results: Even the "Smartest" Robots Struggle

They tested the world's most advanced AI models (like Claude 4.0 and GPT-5) on these mini-challenges.

• The Reality Check: Even the smartest robots only got about 65% of these simple tasks right.
• The Analogy: It's like giving a brilliant human a test where they have to tie a specific knot or fill out a tax form. Even smart people make mistakes if the instructions are tricky or the interface is confusing. The robots are failing to "read the room" of the website.

4. The Fix: Training with "Video Games"

The authors wanted to see if they could teach open-source robots (which are usually weaker) to get better. They used two training methods:

1. Supervised Fine-Tuning (SFT): Showing the robot thousands of examples of humans successfully doing these tasks, like showing a student a solved math problem.
2. Reinforcement Learning with Verifiable Rewards (RLVR): This is like a video game. They let the robot try the task. If it succeeds, it gets a "point" (reward). If it fails, it gets zero points. The robot learns by playing thousands of games, realizing, "Oh, I clicked the wrong button last time, I shouldn't do that again."

The Outcome:
By using this "video game" training method on synthetic (fake but realistic) websites, their open-source model jumped from a low score to 52.3%. This is impressive because it beat many of the expensive, closed-source "super-brains" on these specific tasks.

5. Why This Matters

The paper concludes that if you want a robot to be good at the big, complex jobs (like booking that vacation), you first have to make sure it is good at the small, boring jobs (like clicking the right date).

They found that a robot's ability to handle these tiny, specific subtasks is a very strong predictor of how well it will handle the big, complex tasks. If a robot can't navigate a dropdown menu, it probably won't be able to plan a trip.

In short: The authors built a safe, time-frozen playground to test how well robots can handle the tiny, tricky details of using a website. They found that even the best robots are bad at these details, but they can be trained to get much better by playing "video games" where they get points for doing it right.

Technical Summary

Technical Summary: WARC-Bench

Problem Definition

The paper identifies a critical gap in the evaluation of autonomous web agents. Existing benchmarks typically focus on two extremes: single-step visual grounding (mapping text to pixel coordinates) or complex, long-horizon web navigation tasks (e.g., purchasing an item on Amazon). The intermediate level of complexity—GUI subtasks—remains largely unaddressed. These subtasks are short-horizon workflows consisting of multiple primitive UI actions (e.g., selecting a specific date in a date picker, scrolling a container to extract data, or interacting with a dropdown) that constitute a single functional instruction within a larger browsing session.

The authors argue that mastering these subtasks is essential for robust web planning and navigation, yet current benchmarks fail to evaluate this capability effectively. Furthermore, existing long-horizon benchmarks often lack the diversity and stress-testing capabilities required to isolate and diagnose failures in specific interaction components.

Methodology

WARC-Bench Construction

The authors introduce WARC-Bench (Web Archive Benchmark), a novel benchmark designed to evaluate multimodal AI agents on subtask execution.

• Environment: The benchmark utilizes Web ARChive (WARC) files to create sandboxed, interactive web environments. These files capture the complete state of a website (HTML, CSS, JavaScript, images, HTTP headers) allowing for high-fidelity replay within a Chromium-based browser (via Playwright). This approach ensures task isolation, prevents real-world write operations, and allows for rapid, scalable evaluation.
• Dataset Composition: The benchmark comprises 438 tasks divided into:
  • Test Set: 200 tasks derived exclusively from real-world websites (e.g., GitHub, Zoho Desk, Kaiser Permanente).
  • Development Set: 238 tasks (60 real, 178 synthetic).
  • Training Set: 1,059 synthetic tasks generated via an LLM-based pipeline to create diverse UI widgets and task goals.
• Task Categories: Tasks cover 15 categories including date pickers, form filling, table manipulation, drag-and-drop, and data extraction.
• Evaluation: Each task includes a deterministic, programmatic reward function (JS, URL, String, or JSON matchers) to verify task completion independent of the agent's specific trajectory.

Agent Design and Training

The authors propose the Subtask Vision Agent (SVA), a generalist agent design that relies solely on screenshots for observation, avoiding reliance on accessibility trees or HTML DOM which can be redundant or exceed context limits.

• Action Space: The agent operates on a minimal API of eight actions: click, complete, drag_and_release, hover, key_press, scroll, type, and wait.
• Training Strategy: To improve open-source models (specifically the Qwen2.5-VL family), the authors employ a two-stage training pipeline:
  1. Supervised Fine-Tuning (SFT): Distillation from trajectories generated by strong "teacher" models (including frontier models) on a dataset of ~12k unique trajectories.
  2. Reinforcement Learning with Verifiable Rewards (RLVR): Applying Proximal Policy Optimization (PPO) on top of SFT checkpoints using the synthetic WARC-Bench training set. The agent receives a reward of 10 for successful trajectories and 0 for failures or timeouts.

Key Results

Benchmark Performance

WARC-Bench proves challenging even for state-of-the-art frontier models:

• Anthropic's Claude-4.0-Sonnet achieved the highest success rate among evaluated models at 64.8% on the test set.
• OpenAI's GPT-5 achieved 51.3%.
• Open-source models (e.g., Qwen2.5-VL-72B) initially lagged significantly at 37.3%.

Impact of Training Techniques

The proposed training pipeline significantly narrowed the gap between open-source and closed-source models:

• SFT: Improved the Qwen2.5-VL-72B model from 37.3% to 48.3%.
• RLVR: Further improved the Qwen2.5-VL-72B model to 52.3%, outperforming many frontier closed-source models and establishing a new state-of-the-art for open-source agents on this benchmark.
• Efficiency: The RLVR-trained models demonstrated improved exploration (increased scrolling), reduced redundant clicks, and shorter trajectory lengths (approx. 0.94 fewer steps per task) compared to SFT-only baselines.

Comparative Analysis

• Correlation with Long-Horizon Tasks: Unlike benchmarks like ScreenSpot V2 or MiniWoB++, performance on WARC-Bench correlates more strongly with long-horizon navigation capabilities (measured on WebArena).
• Hierarchical Agents: The paper demonstrates that decomposing long-horizon tasks into subtasks using a Planner (e.g., Claude-4.0-Sonnet) and an Executor (e.g., the trained Qwen2.5-VL-72B-RLVR) significantly improves performance on WebArena-Lite (from 16.17% to 24.63% for Qwen-based agents), validating the utility of specialized subtask models.

Significance and Claims

The paper claims that WARC-Bench fills a critical gap in the landscape of GUI agent research by providing a high-fidelity, scalable, and sandboxed environment for evaluating short-horizon subtasks. The authors assert that:

1. Subtask Mastery is Prerequisite: Successfully executing subtasks is a fundamental capability required for robust long-horizon web navigation, a capability not extensively evaluated by existing benchmarks.
2. Training Efficacy: Synthetic data combined with SFT and RLVR can enable open-source models to achieve competitive, and in some cases superior, performance to frontier closed-source models on realistic web interactions.
3. Diagnostic Value: The benchmark serves as a stress test for specific agent capabilities (e.g., visual grounding on complex widgets, handling dynamic UIs like date pickers) that often cause failures in larger systems.

The work concludes that WARC-Bench offers a unique pathway to advance generalist computer-use agents by systematically improving the intermediate subtask execution layer.