🤖 AI

AliyunConsoleAgent: Training Web Agents in Real-World Cloud Environments via Distillation and Reinforcement Learning

The paper introduces AliyunConsoleAgent, a cost-effective web agent framework that achieves near-state-of-the-art performance in verifying cloud console documentation through a two-stage training paradigm combining supervised fine-tuning on distilled trajectories and reinforcement learning with a robust reward system in real-world environments.

Original authors: Bojie Rong, Zheyu Shen, Qiaoping Wang, Pengfei Kang, Yang Xu, Yawen Wei, Hanyu Wu, Zhi Zhao, Leihao Pei, Linquan Jiang

Published 2026-06-09

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bojie Rong, Zheyu Shen, Qiaoping Wang, Pengfei Kang, Yang Xu, Yawen Wei, Hanyu Wu, Zhi Zhao, Leihao Pei, Linquan Jiang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a massive, constantly changing theme park (the Cloud Console). Every day, the park adds new rides, changes ticket booths, and updates safety rules. Meanwhile, the guidebooks (the Documentation) are written by a different team that updates them slowly by hand.

The Problem:
Because the park changes so fast, the guidebooks quickly become outdated. A step in the book might say, "Click the blue button," but in the live park, that button is now red, or it's been replaced by a touchscreen.
Checking every single instruction in the guidebook against the live park is a nightmare. It would take a human team millions of hours a year to do it, and currently, they only check about 1% of the instructions. If they don't check, customers get lost and frustrated.

The Old Solution (and why it failed):
The company tried hiring "super-intelligent robots" (Frontier Proprietary Models) to do the checking. These robots are incredibly smart and can read the guidebooks and navigate the park perfectly. However, they are expensive (like hiring a team of PhDs for every single check) and risky (you have to show them your private park maps, which violates security rules). You can't afford to use them for the millions of checks needed.

The New Solution: AliyunConsoleAgent
The authors built their own "trainable robot" (a 32-billion parameter AI model) that is cheaper and safer to run on their own servers. But a standard robot isn't smart enough to handle a chaotic, changing theme park. So, they used a two-step training method:

1. The "Shadowing" Phase (Supervised Fine-Tuning)

First, they took the super-intelligent robots and made the new robot shadow them.

The Analogy: Imagine a master chef (the super-robot) cooking a complex dish. The new robot watches the master, memorizing every chop, stir, and seasoning step.
The Result: The new robot learns the basics of how to navigate the park and follow instructions. It gets pretty good, but it's still just copying. If the park changes slightly, it gets confused because it doesn't truly understand the goal, it just remembers the steps.

2. The "Trial and Error" Phase (Reinforcement Learning)

Next, they let the robot loose in a simulated version of the park to learn by doing.

The Challenge: In a real cloud environment, the robot often fails not because it's stupid, but because the "park" isn't ready. For example, it tries to delete a server, but the server doesn't exist yet. If the robot gets punished for this, it learns the wrong lesson (that it's bad at deleting servers) instead of the right one (that it needs to build the server first).
The Fix (The High-Determinism Rollout): The team built a special "training ground" using Terraform (a tool that builds infrastructure like Lego). Before the robot tries to do a task, this system automatically builds the exact prerequisites (like building the server, setting up the network) so the robot can succeed if it makes the right moves.
The Reward System: Instead of a human grading the robot, they use a Dual-Channel Judge:
1. The Rule-Checker: It looks at the official "audit logs" (the park's security camera footage) to see if the action actually happened. Did the server get deleted? Yes/No. This is 100% objective.
2. The Panel of Judges: If there's no clear log, two other AI models act as judges. They only give a "pass" if they both agree the robot succeeded. This prevents the robot from "cheating" or tricking a single judge.

The Result

After this training, the robot evolved from a mindless follower into an autonomous problem solver.

Before: If the guide said "Turn off auto-renewal" but the switch was already off, the robot would just stop and say "I can't do it."
After: The robot thinks, "Wait, I can't turn it off if it's already off. I need to turn it on first so I can then turn it off to prove I did it." It figured out the logic on its own.

The Bottom Line:

Performance: Their new robot (AliyunConsoleAgent-32B) is almost as good as the expensive "super-robots" (within 1.8% difference).
Cost: It costs 92% less to run.
Impact: They used this system to audit over 54,000 instructions and found nearly 4,400 real errors that product teams fixed.

In short, they taught a local, affordable robot to think like a genius by having it shadow a genius, and then letting it practice in a perfectly prepared training gym where it could learn from its mistakes without being punished for things outside its control.

Technical Summary: AliyunConsoleAgent

1. Problem Statement

Cloud platforms face a structural "documentation drift" problem where user interface (UI) elements evolve rapidly under short release cycles, while accompanying documentation is manually maintained. This divergence leads to procedures that no longer match the live console. Verifying that documentation accurately reflects the current state requires an estimated 4 million recurring inspections annually, yet manual coverage remains below 1%.

While frontier proprietary models can automate this "Execution-as-Verification" task, their deployment at scale is hindered by:

Prohibitive Costs: High per-call inference costs prevent full-coverage auditing.
Data Privacy: Transmitting sensitive cloud metadata to external services poses compliance risks.
Environmental Noise: Real-world cloud tasks have strong resource dependencies (e.g., specific ECS instance types). Without proper pre-provisioning, environmental failures (missing resources) are indistinguishable from agent errors, polluting reward signals and preventing Reinforcement Learning (RL) convergence.

2. Methodology

The authors propose AliyunConsoleAgent, a framework utilizing a two-stage training paradigm (Supervised Fine-Tuning + Reinforcement Learning) within a high-determinism rollout environment.

A. High-Determinism Rollout Environment

To isolate environmental noise from the training signal, the authors constructed a four-layer architecture:

Account Pool Management: Maintains isolated test accounts with automated credential rotation to prevent cross-task interference.
Sandbox Execution: Uses containerized environments (Alibaba Cloud ACK) for task-level isolation.
Resource META Offline Provisioning: For each document, a "Resource META" tuple defines dependencies. A pipeline (ResourceCoder) generates Terraform templates to pre-provision necessary resources (VPCs, instances, etc.) from scratch with randomized naming, ensuring cross-account portability.
Runtime On-demand Provisioning: An LLM-based agent (ResourceCoder) dynamically provisions missing resources if the agent encounters unexpected blocks (e.g., empty resource lists), injecting structured reports back into the context.

This system resolves the speed-accuracy tradeoff, enabling parallel execution with near-perfect resource readiness, lifting success rates from ~34% (naive baselines) to ~84%.

B. Training Paradigm

The model (based on Qwen3-VL-32B) undergoes two stages:

Supervised Fine-Tuning (SFT):
- Data Construction: Combines distilled trajectories from frontier models (filtered for success and step-level quality) with autonomous self-exploration data (CRUD tasks).
- Format: Enforces a structured ReAct paradigm where the model generates explicit "Thought" traces before actions to assess progress and identify sub-goals.
Reinforcement Learning (GRPO):
- Algorithm: Uses Group Relative Policy Optimization (GRPO) to evolve the model from instruction following to autonomous decision-making.
- Dual-Channel Outcome Reward Model (ORM):
  - Channel 1 (Rule-Based): Queries ActionTrail (audit logs) for deterministic success criteria, providing ground-truth binary rewards.
  - Channel 2 (LLM-as-Judge Ensemble): For tasks lacking definitive API events, an ensemble of two strong LLMs acts as judges. Consensus is required; disagreements result in invalid labels to prevent reward hacking.
- Normalization: Applies two-layer advantage normalization (per-task and global) to handle heterogeneous task difficulties.

3. Key Contributions

High-Determinism Rollout Environment: A novel infrastructure combining Terraform-based pre-provisioning and LLM-driven on-demand provisioning with audit-log verification. This effectively isolates environmental noise, enabling stable RL training in real production clouds.
Cost-Effective Private Agent Training: A successful SFT+GRPO pipeline that produces a 32B parameter model capable of approaching the performance of frontier proprietary models at 92% lower inference cost.
Production-Scale Verification: Deployment of a verification framework that audited 54,000+ procedures, identifying 4,399 confirmed defects. The trained agent is validated as a privacy-compliant, drop-in replacement for frontier models.

4. Experimental Results

The model was evaluated on a challenging 278-task benchmark (76 standard, 202 hard) across 12 Alibaba Cloud products. Evaluation was strictly rule-based via ActionTrail audit logs.

Performance:
- AliyunConsoleAgent-32B (SFT+GRPO) achieved a 63.52% mean success rate (pass@1).
- This represents a 20.24 percentage-point (pp) improvement over the base model (43.28%).
- It narrows the gap to the best frontier model (Gemini 3 Pro Preview, 65.34%) to just 1.82 pp. Statistical analysis (bootstrap 95% CI) shows this gap is not statistically significant ( $p > 0.05$ ).
Cost Efficiency:
- The private 32B model costs approximately 0.56 CNY per task.
- Compared to Gemini 3 Pro Preview (~7.00 CNY/task), this is a 92% cost reduction.
Ablation:
- SFT contributed +13.61 pp over the base model.
- GRPO contributed an additional +6.63 pp over SFT, with the improvement being statistically significant.
Qualitative Improvements: The RL-trained model demonstrated "Precondition Construction" (e.g., enabling a feature before disabling it) and "Adaptive Plan Adjustment" (switching to alternative UI paths when documented steps fail), capabilities absent in the SFT-only model.

5. Significance and Claims

The paper claims that AliyunConsoleAgent demonstrates the feasibility of training high-performance web agents entirely within real-world, privacy-sensitive cloud environments without relying on expensive frontier APIs.

Bridging the Gap: It proves that a 32B open-source model, when trained with high-quality distillation and robust RL in a noise-isolated environment, can match the performance of much larger proprietary models on complex, state-dependent tasks.
Scalability: The proposed architecture makes large-scale, periodic documentation verification economically and technically feasible, addressing a critical industry need for maintaining documentation integrity.
Reproducibility: The authors have open-sourced the evaluation benchmark, filtered training data, rollout infrastructure code, and model training code to support further research in real-world cloud agent training.

The authors acknowledge limitations, including the maintenance cost of static Terraform templates, the fragility of DOM-parsing for visual grounding, and the infrastructure expense of running live verification on premium resources. However, they assert that the current system provides a viable, cost-effective path for sustainable, full-scale deployment.