🤖 machine learning

ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents

The paper introduces ClawGUI, an open-source framework that unifies training, evaluation, and deployment of GUI agents by providing a stable RL infrastructure for virtual and physical devices, a standardized benchmarking pipeline, and cross-platform agent deployment, ultimately enabling the ClawGUI-2B model to outperform existing baselines in mobile GUI tasks.

Original authors: Fei Tang, Zhiqiong Lu, Boxuan Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fei Tang, Zhiqiong Lu, Boxuan Zhang, Weiming Lu, Jun Xiao, Yueting Zhuang, Yongliang Shen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot to use your smartphone. You don't want to give it a secret "cheat code" (API) that only the app developers know. Instead, you want the robot to look at the screen, see the buttons, and tap or swipe just like a human does. This is what a "GUI Agent" does.

However, building a robot that can actually learn to do this, be tested fairly, and then live on your phone to help you is incredibly hard. The research community has been stuck in three separate silos: one group builds the training gym, another builds the test track, and a third tries to drive the car, but they never talk to each other.

ClawGUI is the paper that finally builds the entire highway system connecting all three. It's a single, open-source toolkit that handles everything from teaching the robot to driving it in the real world.

Here is the breakdown using simple analogies:

1. The Problem: The "Broken Assembly Line"

Before ClawGUI, the process was like trying to build a car in three different countries:

Training: Researchers tried to teach robots in a video game (simulators), but the game was buggy, and no one shared the game engine.
Testing: Everyone used different rulers to measure speed. One person measured in miles, another in kilometers, and some didn't even write down which ruler they used. You couldn't tell if a new robot was actually faster or just measured differently.
Deployment: Even if a robot learned to drive, it stayed in the lab. It couldn't get into a real car (your phone) or talk to you in a way you're used to (like a chat app).

2. The Solution: The "ClawGUI" Factory

The authors built a unified factory with three main departments that work together seamlessly.

🏭 Department A: The Training Gym (ClawGUI-RL)

The Old Way: Teaching a robot to tap a button was like throwing a dart in the dark. You only knew if you hit the bullseye at the very end of the game. If the robot took 50 steps to get there, it didn't know which of those 50 steps were good.
The ClawGUI Way: They introduced a "Process Reward Model" (PRM). Imagine a coach standing next to the robot, whispering after every single tap: "Good job! That was the right button," or "Oops, you went the wrong way."
The Hardware: They built a system that can run hundreds of virtual phones at once (like a massive server farm of emulators) AND train directly on real physical phones. It's like having a gym where you can practice on a treadmill or on the actual street, and the system automatically swaps out broken treadmills so training never stops.

📏 Department B: The Standardized Test Track (ClawGUI-Eval)

The Problem: In the past, if Paper A said their robot was 90% accurate and Paper B said 88%, you couldn't compare them. Maybe Paper A used a high-resolution photo and Paper B used a blurry one.
The ClawGUI Way: They built a universal referee. They took 6 different famous test tracks (benchmarks) and 11 different robot models. They forced everyone to use the exact same camera settings, the exact same lighting, and the exact same scoring rules.
The Result: They proved that when you use the same rules, you can reproduce 95.8% of previous results. It's like finally agreeing that "100 meters" means the same thing for every runner in the world.

🚗 Department C: The Real-World Driver (ClawGUI-Agent)

The Problem: Most research robots are stuck in a lab. They can't talk to you on WhatsApp or WeChat, and they forget your preferences after every task.
The ClawGUI Way: They built a bridge that lets the robot live in your chat apps (like Telegram, Slack, or DingTalk).
- Hybrid Driving: If an app has a "secret command" (CLI), the robot uses it for speed. If not, it switches to "visual driving" (GUI) and taps the screen. It's like a driver who knows how to use the car's autopilot but can also manually steer if the road gets weird.
- Memory: The robot remembers you. If you always order coffee at 8 AM, it learns that. It stores these habits in a "personal memory bank" so it gets smarter about you over time.

3. The Proof: The "ClawGUI-2B" Robot

To prove their system works, they built a robot called ClawGUI-2B.

It's a relatively small robot (2 billion "brain cells").
They trained it entirely inside their new system.
The Result: It beat much larger, more expensive robots (some with 72 billion brain cells) on a difficult mobile phone test.
The Lesson: A small robot trained with a great system (good coach, fair test, real-world practice) beats a giant robot trained with a broken system.

The Big Picture

ClawGUI is the "Operating System" for the next generation of AI helpers.

It stops researchers from reinventing the wheel.
It stops them from fudging the numbers.
It finally lets these AI helpers leave the lab and actually help you on your phone, remembering your habits and navigating any app you throw at it.

It's the difference between teaching a robot to play a video game in a simulator and teaching it to actually drive your car to the grocery store.

1. Problem Statement

The paper identifies that progress in Graphical User Interface (GUI) agents is currently bottlenecked not by model capacity, but by the lack of a coherent full-stack infrastructure. The community faces three critical gaps:

Closed Training Ecosystems: Existing online Reinforcement Learning (RL) systems are largely closed-source, tied to specific emulator sandboxes, and lack support for training on real physical devices. This prevents reproducibility and limits the ability to train agents for real-world deployment.
Misaligned Evaluation: Benchmark results across different papers are rarely comparable due to undocumented variations in prompt formatting, coordinate normalization, image resolution, and sampling configurations. There is no shared baseline to measure true progress.
Broken Deployment Loop: Trained agents rarely reach end-users. Existing solutions are either CLI-based (lacking visual grounding and universal coverage) or isolated demos that lack persistent personalization and cross-platform integration (Android, HarmonyOS, iOS).

2. Methodology: The ClawGUI Framework

ClawGUI is an open-source framework designed to close these gaps through three tightly integrated modules: ClawGUI-RL, ClawGUI-Eval, and ClawGUI-Agent.

A. ClawGUI-RL: Scalable Online RL Training

This module provides the first open-source infrastructure for training GUI agents on both virtual and physical devices.

Environment Management: It abstracts backends to support parallel Docker-based Android emulators and real physical devices. It features a Rollout Manager with health checks, crash recovery, and spare server rotation to maintain stability during long training runs.
Reward Design: To address the sparsity of rewards in long-horizon tasks, ClawGUI-RL employs a two-level reward system:
1. Binary Outcome Reward: A final score (0 or 1) based on task success.
2. Dense Step-Level Reward: A Process Reward Model (PRM) that evaluates every intermediate step. The PRM analyzes the previous screenshot, current screenshot, and action history to determine if a step contributes to the goal.
RL Algorithms: It integrates GiGPO (Grouped Importance-weighted Policy Optimization) alongside standard GRPO. Unlike GRPO, which assigns a uniform advantage to an entire trajectory, GiGPO uses an anchor-state grouping mechanism. It clusters steps encountering the same intermediate state across different rollouts to estimate micro-relative advantages, enabling fine-grained credit assignment without a learned value network.

B. ClawGUI-Eval: Reproducible Evaluation

This module enforces a strict, standardized three-stage pipeline to ensure fair comparison across models and benchmarks.

Pipeline Architecture:
1. Infer: Generates raw predictions using local GPUs (via Transformers) or remote APIs.
2. Judge: Parses outputs using benchmark-specific judges (e.g., point-in-box for grounding, polygon/refusal-aware for OSWorld-G).
3. Metric: Aggregates results into accuracy scores with granular breakdowns.
Standardization: All evaluation choices (prompts, normalization, resolution) are "pinned" per model to eliminate configuration drift. It supports 6 benchmarks (including ScreenSpot-Pro, AndroidControl, MobileWorld) and 11+ models.

C. ClawGUI-Agent: Real-Device Deployment

This module bridges the gap between research and production, deploying trained agents to real users.

Hybrid Control: It combines CLI efficiency (for operations with programmatic interfaces) with GUI control (for visual interaction with any app), ensuring broad coverage.
Personalized Memory: A persistent memory system extracts structured facts (contacts, habits, preferences) from interactions, stores them as vector embeddings, and retrieves top- $k$ relevant memories to adapt the agent to individual users over time.
Cross-Platform Support: It connects to Android, HarmonyOS, and iOS via 12+ chat platforms (e.g., Feishu, DingTalk, Telegram), allowing remote or local control via natural language.

3. Key Contributions

Unified Open-Source Framework: The first system integrating scalable online RL training, standardized evaluation, and real-device deployment into a single pipeline.
ClawGUI-RL Infrastructure: Validated support for parallel virtual environments and real physical device training, utilizing GiGPO and PRMs for dense supervision.
ClawGUI-Eval Standardization: A reproducible evaluation suite covering 6 benchmarks and 11+ models, achieving a 95.8% reproduction rate against official baselines.
ClawGUI-Agent: A production-ready deployment system with hybrid CLI-GUI control and persistent personalized memory.
ClawGUI-2B Model: A 2B parameter agent trained end-to-end within the framework, demonstrating the efficacy of the infrastructure.

4. Experimental Results

Performance on MobileWorld (GUI-Only):
- ClawGUI-2B achieved a 17.1% Success Rate (SR).
- This outperforms the same-scale baseline MAI-UI-2B (11.1%) by 6.0 absolute points (a 54% relative improvement), proving the value of the RL infrastructure.
- It significantly surpasses much larger untrained models, such as Qwen3-VL-32B (11.9%) and UI-Venus-72B (16.4%).
Impact of Dense Rewards:
- Replacing episode-level GRPO with step-level GiGPO improved SR from 14.5% to 17.1% (a 17.9% relative gain), confirming that fine-grained credit assignment is critical for GUI agents.
Evaluation Reproducibility:
- ClawGUI-Eval achieved a 95.8% reproduction rate (46/48 cells) against official baselines.
- The few failures were attributed to undisclosed configurations in closed-source models, validating that evaluation discrepancies are an infrastructure issue, not a fundamental limitation.

5. Significance

ClawGUI represents a paradigm shift in GUI agent research by moving from isolated modeling experiments to a systems-engineering approach.

Democratization: By open-sourcing the full stack (training, eval, deployment), it allows researchers to reproduce results and build upon a stable foundation rather than reinventing infrastructure.
Real-World Viability: It proves that small, well-trained models (2B parameters) can outperform massive untrained models when supported by robust RL infrastructure and dense reward signals.
Future Direction: It lays the groundwork for "On-Device, Always-Present System Agents" that can operate autonomously across diverse operating systems with persistent memory, bridging the gap between academic benchmarks and practical, personalized digital assistants.