UI-Venus-1.5 Technical Report

The paper introduces UI-Venus-1.5, a unified end-to-end GUI agent family featuring 2B, 8B, and 30B-A3B variants that leverage mid-training, online reinforcement learning, and model merging to achieve state-of-the-art performance on diverse benchmarks and robust real-world navigation across Chinese mobile apps.

The Gist

Imagine you have a very smart, but slightly clumsy, digital assistant. You ask it to "Book a ticket to Beijing," and instead of just doing it, it gets confused, clicks the wrong button, or tries to type in a box that doesn't exist. This is the current state of many "GUI Agents" (computer programs that try to use apps and websites like humans do).

The UI-Venus-1.5 paper from Ant Group is like a report on how they took that clumsy assistant, gave it a massive education, trained it in a high-tech gym, and turned it into a world-class personal shopper and travel agent.

Here is the story of how they did it, explained with simple analogies:

1. The Problem: The "Smart but Clueless" Assistant

Before this update, AI models were great at chatting but terrible at actually doing things on a screen. They knew what a "button" looked like in a textbook, but when faced with a real, messy app on a phone, they got lost. They were like a student who memorized the map of a city but has never actually walked the streets.

2. The Solution: The Three-Step Training Camp

The team didn't just tweak the model; they built a completely new training pipeline with three distinct phases. Think of it as a three-year university degree for a robot.

Phase 1: The "Mid-Training" Boot Camp (The Library)

• The Analogy: Imagine your assistant is a new employee. Before they can do their job, you don't just throw them into the office. You send them to a library for a month to read every manual, every user guide, and every screenshot of every app imaginable.
• What they did: They fed the model 10 billion tokens of data from over 30 different datasets. This wasn't just random chat; it was specific "GUI" (Graphical User Interface) data.
• The Result: The model stopped guessing what a "search bar" or a "submit button" was. It learned the language of screens. It went from "I think that's a button" to "I know exactly what that button does and where it lives."

Phase 2: The "Offline" Simulation (The Flight Simulator)

• The Analogy: Now that the assistant knows the theory, they need to practice without crashing a real plane. They put the assistant in a flight simulator. They run thousands of scenarios where the assistant tries to book a flight, but if they make a mistake, the simulator says, "No, that's wrong, try again," and gives them a score.
• What they did: They used Offline Reinforcement Learning. They took existing records of humans using apps and taught the AI to mimic those successful paths. They also taught the AI a crucial new skill: Refusal. If you ask the AI to click a button that doesn't exist, instead of hallucinating and clicking the wrong thing, it learns to say, "I can't find that button."
• The Result: The model became much more accurate at finding specific items on a screen (Grounding) and stopped making up fake buttons.

Phase 3: The "Online" Real-World Gym (The Live Fire Drill)

• The Analogy: Simulators are great, but real life is messy. The internet changes, apps update, and things break. So, they put the assistant in a "Live Fire Drill." They gave it thousands of real phones and computers in the cloud and told it: "Go try to do these tasks. If you fail, learn from it immediately."
• What they did: This is Online Reinforcement Learning. The model actually interacted with real apps, saw what happened, and adjusted its brain instantly. They built a massive "Device-as-a-Service" system (like a giant warehouse of thousands of virtual phones) to let the AI practice millions of times a day.
• The Result: The model learned to handle long, complex tasks (like "Find a vegetarian lasagna recipe under 600 calories") without getting lost halfway through. It learned to recover from mistakes, just like a human would.

3. The Magic Trick: The "Model Merge" (The All-Star Team)

Usually, to make a robot good at everything, you have to train three different robots: one for mobile phones, one for websites, and one for finding specific buttons. Then you have to switch between them.

• The Analogy: Imagine you have a soccer coach, a basketball coach, and a tennis coach. Instead of hiring three different people, the UI-Venus team took the best moves from all three coaches and blended them into one single super-coach.
• What they did: They trained three specialized models, then used a "Model Merging" technique (specifically called TIES-Merge) to combine them into one single brain.
• The Result: You now have one AI that can handle your phone, your laptop, and complex websites seamlessly. It's not just a "jack of all trades, master of none"; it's a master of all three.

4. The Results: A New Champion

The paper shows that this new "Super Assistant" (UI-Venus-1.5) is crushing the competition:

• On Mobile Apps: It successfully completed tasks on Android phones at a 77.6% success rate, beating all previous models.
• On Websites: It navigated complex websites with 76.0% accuracy.
• On Precision: It can find tiny icons on a screen with incredible accuracy, even in professional software like CAD tools.

Why Should You Care?

Think about the apps you use every day: booking a train ticket, buying groceries, or managing your bank account. Doing these on a phone can be tedious.

UI-Venus-1.5 is the first step toward an AI that doesn't just talk to you about doing these things, but actually does them for you. It's like having a personal assistant who can look at your screen, understand what you want, and click the right buttons to get it done, even if the app is in Chinese or the layout is confusing.

In short: They took a smart but clumsy robot, gave it a massive education, trained it in a real-world gym, and fused its skills into one super-brain. The result is an AI that can finally navigate our digital world as well as a human can.

Technical Summary

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have advanced the field of GUI Agents, significant challenges remain in achieving a balance between broad generality and consistent, high-performance task execution in real-world scenarios.

• Limitations of Previous Approaches: Existing methods often rely on rigid APIs or suffer from a mismatch between step-level accuracy (individual actions) and trace-level accuracy (full task completion).
• Domain Fragmentation: Previous models often required separate architectures or checkpoints for different domains (e.g., grounding, web navigation, mobile navigation), complicating deployment.
• Data and Reward Sparsity: Standard pre-training lacks fine-grained GUI structural understanding, and offline training often fails to capture the stochasticity and long-horizon dynamics of real-world environments, leading to brittle policies.

2. Methodology

UI-Venus-1.5 is a unified, end-to-end multimodal agent designed to bridge high-level user instructions with concrete GUI interactions. The system is built upon the Qwen3-VL architecture and employs a four-stage training pipeline:

A. Mid-Training (Knowledge Injection)

Before reinforcement learning, the model undergoes a comprehensive mid-training stage to bridge the semantic gap between general vision and GUI-specific structures.

• Data Scale: Utilizes 10 billion tokens aggregated from 30+ diverse datasets (including Mind2Web, ShowUI, AITW).
• Objectives: Covers navigation, grounding, sequential reasoning (Chain-of-Thought), GUI-VQA, and fine-grained perception.
• Data Refinement: Employs an iterative "Teacher-Student" loop where a high-capacity MLLM (Qwen3-VL-235B) scores and refines trajectories. Low-quality traces are rewritten or discarded, increasing high-fidelity sample proportion from 69.7% to 89.7%.
• Real-Device Loop: A Data-as-a-Service (DaaS) system generates 30,000+ verified interaction trajectories from real cloud-hosted devices, improving trajectory generation success rates from 17.9% to >70%.

B. Offline Reinforcement Learning (Offline-RL)

The model is fine-tuned on specific domains (Grounding, Mobile, Web) using a decoupled reward system:

• Reward Components: Includes format rewards (XML structure), action-type rewards, and coordinate/content rewards.
• Refusal Capability: Introduces a "refusal" mechanism where the model outputs [-1, -1] for non-existent elements, reducing hallucinations.
• Observation: While step-level accuracy improves, trace-level accuracy often peaks and declines due to the sparsity of rewards in offline data, necessitating the next stage.

C. Online Reinforcement Learning (Online-RL)

To address the step-trace mismatch and dynamic environments, the model engages in online learning via a Device-as-a-Service (DaaS) infrastructure.

• Infrastructure: A high-concurrency system (Group Control Gateway) managing thousands of heterogeneous devices (Android, Chrome, Linux) with millisecond-level scheduling.
• Algorithm: Uses Group Relative Policy Optimization (GRPO). It evaluates full trajectories rather than individual steps, assigning advantages based on relative group performance to handle sparse rewards.
• Reward Design: Includes task completion rewards, action penalties for invalid syntax, and a length decay coefficient to encourage efficient paths.
• Stability: Employs Adaptive KL constraints and Annealed Entropy regularization to prevent policy collapse and encourage exploration.

D. Model Merging (Unification)

Instead of training a single massive multi-task model from scratch, the team trains specialized models for Grounding, Web, and Mobile and merges them into a single checkpoint.

• Technique: Uses TIES-Merge (Trim, Elect, Intersect, Merge) rather than simple linear averaging. This prunes conflicting parameter updates and resolves sign conflicts, minimizing performance regression while unifying capabilities.
• Result: A single end-to-end model (UI-Venus-1.5) that handles all domains without needing separate inference pipelines.

3. Key Contributions

1. Unified End-to-End Architecture: Successfully merges domain-specific expertise (grounding, web, mobile) into a single model via TIES-Merge, simplifying deployment.
2. Comprehensive Mid-Training: Introduces a 10B-token mid-training stage that establishes foundational GUI semantics, significantly boosting performance before RL.
3. Scaled Online RL with DaaS: Implements a robust infrastructure for real-time interaction with thousands of devices, solving the "step-trace accuracy mismatch" and improving long-horizon navigation.
4. Refusal-Aware Grounding: Enhances model intelligence by training it to reject invalid instructions (hallucination mitigation) rather than forcing coordinate predictions.
5. Chinese Ecosystem Optimization: Specifically optimized for over 40 Chinese mobile apps, enabling practical automation in local contexts (ticketing, shopping, etc.).

4. Experimental Results

UI-Venus-1.5 achieves State-of-the-Art (SOTA) performance across multiple benchmarks, outperforming both general-purpose VLMs and specialized GUI agents.

• GUI Grounding:
  • ScreenSpot-Pro: 69.6% (SOTA), outperforming MAI-UI-32B (67.9%).
  • VenusBench-GD: 75.0% (SOTA).
  • OSWorld-G-R: 76.4% (SOTA).
• Mobile Navigation:
  • AndroidWorld: 77.6% success rate (SOTA), surpassing Mobile-Agent-v3 and MAI-UI-32B by a significant margin.
  • AndroidLab: 68.1% (human-verified), showing a 5.8% improvement over the previous UI-Venus-1.0-72B model.
  • VenusBench-Mobile: 21.5% (SOTA), a 6.1% absolute gain over the previous best.
• Web Navigation:
  • WebVoyager: 76.0% success rate, closely matching leading baselines like OpenAI-CUA and Holo2-30B.
• Efficiency: The 8B variant of UI-Venus-1.5 often outperforms the 72B variants of previous generations, demonstrating high training efficiency.

5. Significance

• Practical Deployment: By unifying grounding and navigation into a single model and optimizing for real-world Chinese apps, UI-Venus-1.5 moves GUI agents from academic benchmarks to practical, daily-use digital assistants.
• Training Paradigm Shift: The report validates that a combination of Mid-Training (for structural knowledge) + Online-RL (for dynamic adaptation) + Model Merging (for unification) is a superior strategy for building robust, general-purpose GUI agents compared to pure SFT or isolated offline RL.
• Infrastructure Innovation: The DaaS framework demonstrates how to scale RL training for GUI agents across heterogeneous devices, a critical bottleneck in the field.

In conclusion, UI-Venus-1.5 represents a major leap forward in autonomous GUI interaction, offering a scalable, unified, and highly performant solution for automating complex digital tasks across mobile and web platforms.