💬 NLP

OpenForgeRL: Train Harness-native Agents in Any Environment

OpenForgeRL is an open-source framework that enables the end-to-end training of harness-native AI agents in diverse environments by decoupling inference from training via a lightweight proxy and Kubernetes orchestrator, achieving state-of-the-art performance on complex tool and GUI benchmarks while providing insights into how harness choices and reinforcement learning shape agent behavior.

Original authors: Xiao Yu, Baolin Peng, Ruize Xu, Hao Zou, Qianhui Wu, Hao Cheng, Wenlin Yao, Nikhil Singh, Zhou Yu, Jianfeng Gao

Published 2026-08-10

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiao Yu, Baolin Peng, Ruize Xu, Hao Zou, Qianhui Wu, Hao Cheng, Wenlin Yao, Nikhil Singh, Zhou Yu, Jianfeng Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers don't just answer questions but actually do things for you. They can write code, book flights, or organize your digital files. To make this happen, scientists have built "AI agents"—smart programs that can think, plan, and use tools. But here's the tricky part: these agents don't work in a vacuum. They need a "harness," which is like a complex control room or a specialized cockpit. This harness manages the agent's memory, connects it to the internet, and helps it use tools like a calculator or a web browser. Think of the harness as the pilot's seat and the agent as the pilot; you can't just train a pilot in a video game and expect them to fly a real 747. The real plane has specific buttons, emergency protocols, and a cockpit layout that the video game never showed.

For a long time, there was a big problem: the "video games" (simple training environments) didn't match the "real planes" (the complex harnesses used in the real world). Researchers could train agents easily, but when they put them into the real, messy harnesses, the agents often got confused or failed. This paper, titled "OpenForge RL," tackles that exact mismatch. It introduces a new way to train these digital pilots directly inside the real cockpits they will eventually fly, using a system that lets them practice on thousands of different "planes" at the same time without crashing the whole training facility.

The Problem: Training in a Video Game vs. Flying the Real Plane

Imagine you are trying to teach a robot to fix a car. If you train it in a simple video game where the engine is just a red button and the tire is a blue square, the robot learns to press red and grab blue. But when you put that robot in a real garage with a real engine, a real tire, and a million other tools, it freezes. It doesn't know how to handle the real complexity.

This is what has been happening with AI agents. Modern agents are powerful, but they rely on sophisticated "harnesses" (like Claude Code or Codex) to manage their thoughts and tools. These harnesses are like the real cockpit: they have stateful memory (remembering what happened five steps ago), multi-process workflows (running several things at once), and complex tool connections. However, the standard ways researchers train AI (using Reinforcement Learning) usually assume a simple, flat environment. They try to train the agent in a simplified version of the harness, which creates a "train-deploy mismatch." It's like training a pilot on a simulator that doesn't have the real emergency brakes, then expecting them to land a real plane when the brakes fail.

The Solution: OpenForge RL (The Cloud Garage)

The authors of this paper built OpenForge RL, a framework that acts like a massive, cloud-based garage. Instead of trying to force the real, complex harness into a simple training box, OpenForge RL does the opposite: it takes the training box and sends it out to the real harness.

Here is how it works, using a playful analogy:

The Remote Pods (The Cloud Garage): Imagine you have a fleet of tiny, self-driving garages in the cloud. Each garage is a container that holds a specific "harness" (like a specific car model). OpenForge RL uses a "Kubernetes orchestrator" (think of it as a super-efficient garage manager) to spin up thousands of these remote garages instantly.
The Proxy (The Two-Way Mirror): Inside each garage, the AI agent tries to solve a task. A "lightweight proxy" acts like a two-way mirror. It lets the agent talk to the real harness and the real environment (like a real computer or web browser) but secretly records every single move, thought, and tool click.
The Training Loop: The proxy sends these recorded "flight logs" back to the main training brain (which uses standard tools like veRL). The brain learns from these real-world logs, updates the agent's brain, and sends the new version back out to the remote garages to try again.

The magic is that the training happens inside the real harness, in the real environment, but the heavy lifting of managing all those different environments is handled by the cloud. This means researchers can train agents on "ZeroClaw," "OpenClaw," "Codex," or even visual GUI agents (agents that use a mouse and keyboard) all at once, without needing to rewrite their training code for every single new tool.

What They Found: Real Training Works Better

The team tested this system with two types of agents: Claw Agents (which use text and tools to do things like search emails or manage files) and GUI Agents (which look at a screen and use a mouse to click buttons in a web browser or computer).

They trained these agents using only a few hundred to a few thousand tasks—much fewer than some massive models that use millions. The results were impressive:

For Claw Agents: Their model, OpenForge-Claw, scored 31.7 (pass@3) on the ClawEval benchmark and 33.7 on QwenClawBench. This was significantly better than other open-source models of a similar size (around 30 billion parameters). In fact, it performed better than some models that are several times larger.
For GUI Agents: Their model, OpenForge-GUI, reached a score of 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager. This is a big deal because GUI tasks are incredibly hard; the agent has to "see" the screen and figure out where to click. OpenForge-GUI matched or even beat models that were much bigger and had been trained on vastly more data.

The "Harness" Lesson: Not All Cockpits Are Created Equal

One of the most interesting discoveries wasn't just that the training worked, but how different harnesses affected the learning. The authors tested their agents in four different harnesses: ReACT (a simple loop), ZeroClaw (lightweight), OpenClaw, and Codex (very complex).

They found that some harnesses are much harder to learn than others.

Simpler, better-aligned harnesses (like ZeroClaw) allowed the agents to learn faster and perform better.
More complex harnesses (like Codex) were harder to master. The agents struggled more, and while Reinforcement Learning (RL) helped, the gains were smaller compared to the simpler harnesses.

This suggests that the "cockpit" design matters just as much as the pilot's training. A well-designed harness makes the agent smarter and more reliable.

What RL Actually Taught the Agents

The authors also looked closely at what the agents actually learned when they added Reinforcement Learning (RL) on top of basic training (SFT). They found that RL didn't just make the agents "smarter" in a general sense; it specifically improved reliability:

Self-Verification: The agents started checking their own work. For example, if they created a file, they were more likely to read it back to make sure it was correct.
Tool Coverage: They started using a wider variety of tools instead of sticking to just one or two they knew well.
Error Recovery: This was the tricky part. While RL helped the agents recover from small mistakes, error recovery remained weak. Even after training, if an agent made a big mistake, it often couldn't fix it and would just give up. The authors suggest that this specific skill might need special data or different training methods to master.

The Bottom Line

OpenForge RL proves that you don't need to simplify the real world to train AI agents. By using a cloud-based system that decouples the training from the environment, you can train agents directly in the complex, messy, real-world harnesses they will actually use.

The paper suggests that this approach allows researchers to build agents that are not only more capable but also more reliable in their specific jobs. While the agents still struggle with complex error recovery, the ability to train them in the "real cockpit" rather than a "video game" is a major step forward. It means that in the future, we might see AI agents that are truly ready to work alongside us, not just in simulations, but in the actual digital tools we use every day.

Technical Summary: OPENFORGE RL

Problem Statement

Modern AI agents increasingly rely on sophisticated inference harnesses (e.g., Claude Code, Codex, OpenClaw) to manage stateful, multi-turn reasoning, tool use, and context linking with external systems. While these harnesses significantly boost agent performance, they create a critical barrier for open-source research:

Architectural Mismatch: Existing open RL training stacks (e.g., veRL, Slime) assume simple, single-process rollouts. They cannot natively express the stateful, multi-process nature of complex harnesses involving nested tool calls and subagents.
Infrastructure Incompatibility: Running harnesses at scale requires dedicated, containerized environments with specific CPU/memory resources that cannot be co-located on training nodes. Most open frameworks assume rollouts occur locally within the trainer, preventing the use of real-world deployment environments.
Train-Deploy Gap: Consequently, open research often re-implements simplified harnesses for training, creating a mismatch between training conditions and real-world deployment, leaving proprietary systems ahead of open research.

Methodology: OPENFORGE RL

To address these challenges, the authors introduce OPENFORGE RL, an open-source framework designed to train harness-based agents end-to-end in any environment by decoupling training from inference. The system consists of two primary lightweight components:

Lightweight Proxy & Trajectory Reconstruction:
- A proxy server wraps the inference server, intercepting all generation requests issued by the harness containers.
- It routes these requests to standard RL inference engines (e.g., vLLM) and records the prompt-response pairs.
- Upon rollout completion, the proxy collects the terminal reward and reconstructs the full trajectory as a standard training sample: $\tau = \{(s^H_0, a_0, r_0), \dots, (s^H_T, a_T, r_T)\}$ , where $s^H$ represents the harness-processed observation. This allows any harness to run its own arbitrary inference while feeding data into standard RL codebases like veRL.
Kubernetes Orchestrator:
- Following the design of Orchard (Peng et al., 2026), the framework uses a Kubernetes orchestrator to launch each rollout as a remote container on cloud providers (e.g., Microsoft Azure).
- This enables elastic scaling to thousands of concurrent environments without overloading training nodes.
- Robustness Mechanisms: To handle the unreliability of remote rollouts, the system implements wall-clock timeouts (terminating unresponsive jobs rather than blocking the batch) and error handling strategies (discarding trajectories that fail due to infrastructure issues rather than policy errors).
Data Synthesis Pipeline:
- Recognizing the scarcity of RL-ready tasks in non-coding domains (e.g., GUI, daily tool use), the authors built a pipeline to synthesize SFT and RL tasks.
- The pipeline mimics human curation: agents propose task candidates based on reference assets, prune low-quality duplicates, build executable Docker environments with verifiers, and iteratively refine tasks until they pass end-to-end checks.

Key Contributions

Scalable Infrastructure: OPENFORGE RL is the first framework to flexibly pair any inference harness with any environment, connecting real harness ecosystems and remote sandboxes to powerful RL codebases without train-deploy mismatch.
Comprehensive Empirical Study: The authors trained and evaluated agents across diverse settings:
- Claw Agents (Text/Tool-use): Trained on ReACT, ZeroClaw, OpenClaw, and Codex.
- GUI Agents (Computer/Browser-use): Trained on modified Kimi-Agent and Molmo-Web harnesses.
Behavioral Analysis: Enabled by training in real deployment harnesses, the study analyzes how harness choice and RL shape agent behavior, a question previously difficult for open research to address.

Experimental Results

The framework was validated on six benchmarks across tool-use and GUI domains.

Claw Agents (OpenForge-Claw, 30B-A3B MoE):

ClawEval: Achieved 31.7 (pass@3) and 55.9 (pass@3). Note: The paper reports 31.7 as the pass@3 score and 55.9 as the pass@3 score for OpenForge-Claw, while the baseline Qwen3-32B achieved 31.7 on pass@1.
QwenClawBench: Reached 33.7 (pass@1).
MCPAtlas: Achieved 28.1 (pass@1).
Observation: RL training (SFT+RL) provided significant gains over SFT alone, particularly in robustness and average success rates.

GUI Agents (OpenForge-GUI, 8B):

OSWorld-Verified: 37.7 (pass@1).
Online-Mind2Web: 63.0 (pass@1).
WebVoyager: 72.3 (pass@1).
Comparison: Despite using only ~2.5k tasks (compared to 200k+ for MolmoWeb), OpenForge-GUI outperformed similar-sized specialized models (OpenCUA, UI-TARS) and matched or surpassed models several times larger.

Analysis of Harness Choice and RL Impact

The paper provides specific insights into how harnesses and RL affect agent behavior:

Harness Difficulty: Simpler, better-aligned harnesses (e.g., ReACT, ZeroClaw) are easier to learn and yield higher performance than complex, closed-loop harnesses (e.g., Codex) which require more context and show lower gains from RL.
Generalization: Training on a single harness generalizes to unseen harnesses, but training on multiple harnesses simultaneously yields the best cross-harness performance.
RL Capabilities: RL significantly improves agentic reliability, including:
- Self-verification: Agents are more likely to read back their own writes to confirm actions.
- Tool Coverage: Agents utilize a wider set of required tools rather than falling back on generic shells.
- Error Recovery: While RL improves the ability to recover from some errors, this capability remains weak even after training, suggesting it may require dedicated data or methods beyond standard RL.

Significance

The authors claim that OPENFORGE RL lowers the barrier to training and studying agents in the exact harnesses and environments where they are deployed. By enabling end-to-end training with real-world infrastructure, the framework allows researchers to move beyond simplified simulations and directly investigate how inference architectures and reinforcement learning interact to shape agent capabilities. The release of code, data, and models aims to democratize access to harness-native agent training.