🤖 AI

RollArt: Disaggregated Multi-Task Agentic RL Training at Scale

RollArt is a disaggregated multi-task agentic RL system that optimizes training throughput and scalability by mapping pipeline stages to specialized hardware, decoupling trajectory-level interactions to eliminate synchronization bottlenecks, and overlapping rollout with training via asynchronous weight updates, achieving up to 2.05× faster training times on large-scale clusters.

Original authors: Wei Gao, Yuheng Zhao, Tianyuan Wu, Shaopan Xiong, Weixun Wang, Dakai An, Lunxi Cao, Dilxat Muhtar, Zichen Liu, Haizhou Zhao, Ju Huang, Siran Yang, Yongbin Li, Wenbo Su, Jiamang Wang, Lin Qu, Bo Zheng

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wei Gao, Yuheng Zhao, Tianyuan Wu, Shaopan Xiong, Weixun Wang, Dakai An, Lunxi Cao, Dilxat Muhtar, Zichen Liu, Haizhou Zhao, Ju Huang, Siran Yang, Yongbin Li, Wenbo Su, Jiamang Wang, Lin Qu, Bo Zheng, Wei Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but very specific robot (an AI) how to solve complex problems, like writing software code or navigating a website. To do this, you don't just give it a textbook; you let it practice in a real-world simulation. This process is called Agentic Reinforcement Learning.

The paper introduces a new system called ROLLART that acts like a super-efficient traffic controller for this training process. Here is how it works, using simple analogies.

The Problem: The "One-Size-Fits-All" Traffic Jam

Imagine a factory where a single team of workers has to do three very different jobs:

Thinking: Solving hard math problems (requires a fast brain).
Reading: Reading long books quickly (requires fast eyes).
Building: Assembling physical parts (requires strong hands).

In older systems, everyone used the same type of worker for everything. If you assigned a "fast brain" worker to "reading," they were slow. If you assigned a "strong hand" worker to "math," they were slow. Furthermore, if one worker got stuck fixing a broken toy (a "straggler"), the entire factory had to stop and wait for them before moving to the next step. This caused massive delays.

The Solution: ROLLART's Specialized Assembly Line

ROLLART fixes this by breaking the factory into specialized zones and letting different teams work at their own speed.

1. Matching Workers to the Right Tools (Hardware Affinity)

The system realizes that some tasks need computing power (like solving math) and others need memory speed (like reading long texts).

The Old Way: Everyone used the same expensive, heavy-duty computer.
The ROLLART Way: It sends the "math" tasks to super-fast computers and the "reading" tasks to computers with huge, fast memory. It's like sending a chef to a kitchen with a high-powered stove and a librarian to a library with a massive, fast card catalog. They get the job done much faster because they are using the right tool for the job.

2. Letting Workers Move at Their Own Pace (Trajectory-Level Asynchrony)

In the old factory, if one worker took 10 minutes to finish a task while others took 1 minute, the whole line stopped for 10 minutes.

The ROLLART Way: It treats every single practice run (called a "trajectory") as an independent project. If one robot gets stuck on a hard puzzle, the other robots keep working on their own puzzles. The system doesn't wait for the slow one; it just keeps the fast ones moving. Once the slow one finally finishes, it gets scored, and the fast ones move on to new tasks.

3. Hiring "Gig Workers" for Simple Tasks (Serverless Offloading)

After a robot finishes a task, someone has to grade it. Sometimes this is a simple check (like "did the door open?"), and sometimes it's a complex review by another AI.

The Old Way: You kept a team of expensive, high-powered computers sitting idle, waiting to grade these simple tasks. They were expensive to keep "on standby" even when they weren't working.
The ROLLART Way: It uses "gig workers" (serverless cloud computing). You only pay for the grading when it actually happens. If no one is grading, you pay nothing. This frees up the expensive computers to focus on the hard work of teaching the robot.

4. Teaching While Learning (Bounded-Staleness Training)

Usually, the teacher (the training computer) and the student (the robot practicing) take turns. The student practices, stops, waits for the teacher to update their knowledge, and then starts again.

The ROLLART Way: The teacher and student work at the same time. The student keeps practicing with the "yesterday's version" of the teacher's knowledge while the teacher is busy updating the "today's version." The system makes sure the student doesn't get too far behind (using a "staleness bound"), but they never have to stop and wait. It's like a coach shouting new instructions to a runner while the runner is already sprinting, rather than making the runner stop at the starting line every time the coach has a new tip.

The Results

The paper tested this system on a massive scale (using over 3,000 computers at Alibaba).

Speed: It made the training process 1.3 to 2 times faster than previous methods.
Efficiency: It stopped wasting expensive computer power on idle tasks.
Stability: It proved that even when things go wrong (like a computer crashing or a task taking too long), the system keeps running without breaking.

In short, ROLLART is a smart manager that stops the "waiting game" in AI training. It puts the right tasks on the right computers, lets fast workers keep moving even if slow ones lag behind, and uses cheap, on-demand help for simple chores, resulting in a much faster and cheaper way to train advanced AI.

Technical Summary: ROLLART

Problem Statement

Agentic Reinforcement Learning (RL) trains Large Language Models (LLMs) through multi-turn interactions with external environments. This paradigm introduces a complex, heterogeneous workload that mixes four distinct stages: rollout (LLM generation), environment execution, reward evaluation, and training.

Existing systems fail to address the fundamental resource heterogeneity inherent in these stages:

Generation Heterogeneity: LLM generation alternates between compute-bound prefill (dominant in long-context, multi-turn tasks) and bandwidth-bound decoding (dominant in short-turn, long-chain-of-thought tasks). Current systems often assign rollout to a single GPU type, ignoring that compute-optimized GPUs (e.g., H800) excel at prefill while bandwidth-optimized GPUs (e.g., H20) excel at decoding.
Environment Stragglers: Environments are stateful, CPU-bound processes prone to heavy-tailed latency due to container initialization (Docker pulls), network contention, and variable interaction turns. In batched execution, slow environments stall the entire pipeline.
Reward Underutilization: Reward computation is often stateless and bursty. Colocating it with rollout on dedicated GPUs leads to severe underutilization (as low as 7.4%), while coupling it with generation creates resource contention.
Synchronization Overhead: Traditional synchronous training forces rollout workers to idle while waiting for weight synchronization across clusters, creating "dependency bubbles" that waste expensive GPU resources.

Methodology: ROLLART

ROLLART is a distributed system designed for multi-task agentic RL on disaggregated infrastructure. It maps each pipeline stage to the hardware best suited for its specific characteristics and decouples stages to eliminate synchronization bottlenecks.

Core Design Requirements

The system is driven by four empirically derived requirements:

R1 (Hardware-Affinity Mapping): Bind specific tasks or sub-stages to best-fit hardware. Prefill-heavy tasks route to compute-optimized GPUs; decode-heavy tasks route to bandwidth-optimized GPUs; environments run on CPU clusters; and stateless rewards run on serverless infrastructure.
R2 (Trajectory-Level Asynchrony): Decouple generation, environment interaction, and reward scoring at the granularity of individual trajectories. This ensures that slow or failed environments do not block the progress of other trajectories.
R3 (Serverless Offloading): Offload stateless reward computation to serverless platforms (e.g., Function Compute) to eliminate dedicated hot-standby GPUs and enable elastic scaling.
R4 (Bounded-Staleness Training): Execute training and rollout concurrently on separate GPU clusters. A staleness-bounded protocol allows training to proceed on slightly older policies while rollout continues, overlapping weight synchronization with generation to hide latency.

System Architecture

ROLLART consists of three planes:

Resource Plane: A resource manager tracks heterogeneous hardware pools (H800, H20, CPU, Serverless) and binds workers to resources based on user-declared hardware affinities via Python decorators.
Data Plane: Implements the pipeline using Worker and Cluster abstractions. It encapsulates execution logic (e.g., vLLM for generation, Megatron for training) and handles the offloading of stateless tasks to serverless endpoints.
Control Plane: Orchestrates the workflow without user intervention. Key components include:
- LLMProxy: Dispatches generation requests per-trajectory, decoupling clients from inference workers.
- EnvManager: Manages individual environment lifecycles independently, allowing parallel progression.
- SampleBuffer: Buffers scored trajectories for training.
- Asynchronous Training Protocol: A six-step synchronization process (Get Batch → Suspend → Update → Resume → Recompute KV → Train) that overlaps training with rollout. It includes a staleness bound ( $\alpha$ ) to abort trajectories that are too old, ensuring training stability.

Optimizations

Redundant Environment Rollouts: Launches more environments than strictly necessary to mask failures and stragglers; in-flight trajectories are terminated once the target batch is collected.
Prefill-Decoding (PD) Disaggregation: Further refines R1 by routing prefill and decoding phases of the same request to different hardware types (compute vs. bandwidth optimized).
Asynchronous Weight Transfer: Uses a Mooncake-based store to push weights asynchronously. Inference workers pull weights on demand, hiding the cross-cluster transfer latency behind ongoing rollout.

Key Contributions

Disaggregated Infrastructure for Agentic RL: ROLLART is the first system to jointly address prefill/decode divergence, long-tail environment stragglers, stateless reward underutilization, and cross-cluster bandwidth variability at production scale.
Declarative Programming Model: Provides a unified interface for users to declare hardware affinities and offload stateless tasks via decorators, abstracting away the complexity of heterogeneous resource management.
Trajectory-Level Decoupling: Introduces a design where individual trajectories progress independently through generation, environment interaction, and reward, eliminating batch-level synchronization barriers.
Production-Scale Validation: Successfully deployed on an Alibaba cluster with over 3,000 GPUs to train a hundreds-of-billions-parameter Mixture-of-Experts (MoE) model for the Qoder product, running continuously for one week.

Results

Throughput Improvement: ROLLART achieves 2.65–4.58× higher throughput compared to a strengthened synchronous baseline (Sync+).
Training Time Reduction: Compared to various baselines, ROLLART reduces training time by:
- 2.05× over a strengthened synchronous baseline (Sync+).
- 1.35× over a One-off baseline.
- 1.31× over AReaL.
Resource Efficiency: Offloading reward to serverless infrastructure increased reward-GPU utilization from 6% to 88% and halved per-step rollout time.
Scalability: Demonstrated superior scaling efficiency on Qwen3-14B, delivering 1.33–2.08× higher throughput than baselines as cluster size increased from 64 to 128 GPUs.
Staleness Trade-off: An asynchronous bound of $\alpha=1$ was found to balance training speed and model convergence quality effectively.

Significance

The paper claims that ROLLART effectively addresses the "resource-heterogeneity problem" that has limited the scaling of agentic RL. By moving away from monolithic or coarsely decoupled architectures, ROLLART demonstrates that fine-grained, hardware-affinity-aware disaggregation is essential for maximizing throughput in multi-task agentic RL. The system's ability to sustain high throughput and fault tolerance on a 3,000+ GPU cluster validates its stability for production use, offering a blueprint for training large-scale agentic models on heterogeneous infrastructure.