💬 NLP

Envs-FORGE: Frontier-Optimized Reward-Grounded Environment Synthesis for Agent RL

Envs-FORGE is a novel prompting policy that dynamically optimizes environment synthesis for reinforcement learning by converting verifier rewards into per-seed actions via mixed-integer linear programming, thereby generating tailored training environments that significantly improve agent performance across various benchmarks compared to fixed-recipe baselines.

Original authors: Xiaojun Wu, Cehao Yang, Honghao Liu, Xueyuan Lin, Zhichao Shi, Hao Zhou, Xuhui Jiang, Chengjin Xu, Jia Li, Jian Guo

Published 2026-08-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaojun Wu, Cehao Yang, Honghao Liu, Xueyuan Lin, Zhichao Shi, Hao Zhou, Xuhui Jiang, Chengjin Xu, Jia Li, Jian Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to fix a computer or write a piece of code. You can't just hand it a manual; it needs to practice by doing. This is the world of Reinforcement Learning (RL), where an AI agent learns by trying things out, making mistakes, and getting a "reward" (like a high score) when it succeeds. But here's the catch: for the robot to get better, the practice problems it solves need to be just right. If the problems are too easy, the robot gets bored and learns nothing new. If they are too hard, it gets frustrated and gives up. The sweet spot is called the "learning frontier"—the edge of what the robot can currently do, where a little bit of challenge helps it grow the most.

For a long time, scientists have tried to create these practice problems automatically using Large Language Models (LLMs), which are the super-smart AI brains behind chatbots. They used simple, fixed recipes to generate tasks, kind of like a chef using the exact same recipe to cook every single meal, regardless of whether the diner is a toddler or a gourmet chef. This paper introduces a new, smarter way to cook up these practice problems. It's called Envs-FORGE. Instead of using a one-size-fits-all recipe, this new method acts like a personal trainer who watches the robot's current strength, figures out exactly how hard the next challenge should be, and then custom-builds a unique exercise for it. The goal is to make sure every single practice task is perfectly tuned to push the robot just a little bit further than it could go before.

The Problem with "One-Size-Fits-All" Recipes

In the past, when researchers wanted to create new training tasks for these AI agents, they relied on fixed strategies. Imagine a teacher who has a single worksheet and decides to just "make it slightly harder" for every student, no matter who they are. For a student who already aced the test, this new worksheet might still be too easy. For a student who was struggling, it might be impossible.

The paper argues that these fixed methods—like "Self-Instruct" or "Evol-Instruct"—are like that rigid teacher. They apply the same logic to every starting idea (or "seed") without checking if the AI is ready for it. They might make a task harder when it should have been made easier, or they might change the topic entirely when the AI just needed to practice a specific skill deeper. This wastes time and results in training data that isn't very helpful.

Enter Envs-FORGE: The Smart Personal Trainer

The authors of this paper present Envs-FORGE (Frontier-Optimized Reward-Grounded Environment Synthesis). Think of it as a smart system that doesn't just guess what to do next; it calculates it.

Here is how it works, step-by-step:

The Check-Up: First, the system looks at a starting task and asks the current AI agent to try solving it. It counts how often the agent succeeds. This gives it a "pass rate," which is like a report card showing how easy or hard the task is for that specific robot right now.
The Strategy Meeting: Based on that report card, the system considers six different ways to change the task. It can choose to:
- Increase the difficulty (add more constraints or edge cases).
- Reduce the difficulty (simplify it if the robot is stuck).
- Diversify the task (change the context but keep the difficulty similar).
- And for each of these, it can go in-depth (digging deeper into the same topic) or in-breadth (exploring a wider, related topic).
The Math Magic (MILP): The system uses a special kind of math puzzle called a "Mixed-Integer Linear Program" (MILP) to pick the single best combination of changes for that specific task. It's like a coach looking at a player's stats and deciding, "Okay, for this player, we need to increase the difficulty in-depth to hit the perfect learning zone."
The Construction: Once the best strategy is chosen, the system rewrites the entire task package. It doesn't just change the question; it updates the instructions, the data, the solution, the tests, and even the computer environment (like a Docker container) to make sure everything still works together.
The Gold Standard Check: Before the new task is allowed into the training gym, it goes through a strict "Gold Verifier." This is a super-strict test to make sure the new task is runnable, consistent, and actually solvable. If it fails, it's thrown out. Only the perfect ones make the cut.

What They Found

The researchers tested this new method on a powerful AI model called Qwen 3.5 35B. They compared Envs-FORGE against the old fixed recipes (Few-shot, Self-Instruct, and Evol-Instruct) and a baseline model with no extra training.

The results were clear: Envs-FORGE won.

On a benchmark called tb-core, the base model scored 40.0%. The best fixed recipe got 46.8%, but Envs-FORGE soared to 49.2%. That's a 9.2 percentage point improvement over the starting point.
On tb-2.0, the base model scored 23.0%, while Envs-FORGE reached 29.4%, a 6.4 percentage point jump.
Even on a very difficult real-world test called SWE-bench Verified, Envs-FORGE achieved a score of 77.1%, beating the base model's 73.4%.

Crucially, the paper shows that this wasn't because Envs-FORGE used more computer power or generated more data. All the methods generated exactly 100 verified environments for training. The difference was purely in how they chose what those 100 tasks should be. Envs-FORGE just picked better tasks.

Why This Matters

The main takeaway from this paper is that the way we create training data matters just as much as the data itself. By stopping the use of rigid, one-size-fits-all recipes and instead using a smart, data-driven approach to tailor every single task to the AI's current ability, we can help these agents learn faster and better.

The authors suggest that this "frontier-aware" approach—constantly checking where the agent stands and adjusting the challenge to keep it on the edge of its capabilities—is the key to building more capable terminal agents that can handle complex software engineering and system administration tasks. While the study focused on specific benchmarks and model sizes, the results strongly indicate that this method of "smart synthesis" is a significant step forward in teaching AI how to learn.

Technical Summary: Envs-FORGE

Problem Statement

Reinforcement learning (RL) for terminal agents requires executable training environments that provide reliable rewards and possess difficulty levels matched to the current agent policy. Existing approaches often rely on fixed prompting recipes (e.g., few-shot, Self-Instruct, Evol-Instruct) that apply a uniform transformation strategy to every seed task. This "one-size-fits-all" policy fails to account for the specific state of the agent; a seed that is too easy yields weak training signals, while one that is too hard provides no learning signal. Furthermore, fixed recipes cannot dynamically adjust the difficulty trajectory (e.g., creating bridge tasks for seeds currently beyond the agent's capability) or balance skill coverage effectively. The core challenge is determining, for a given seed and current policy, which specific synthesis action will place the resulting environment near the "learning frontier."

Methodology: Envs-FORGE

Envs-FORGE (Frontier-Optimized Reward-Grounded Environment Synthesis) reframes environment synthesis as a reward-grounded action selection problem. Instead of a single fixed prompt, the method employs a dynamic pipeline consisting of four key stages:

1. Frontier-Aware Candidate Scoring

The system first estimates the pass rate ( $\hat{p}_i$ ) of a seed task using verifier rewards from the current agent's rollouts. It then generates six candidate actions for each seed by combining:

Projections: Increase (add constraints/complexity), Reduce (simplify for bridge tasks), and Diversify (change fixtures/requirements while maintaining difficulty).
Evolution Directions: In-depth (deepen existing skills) and In-breadth (expand to related skills).

Candidates are scored based on a "frontier score" ( $F_{i,a,d}$ ), which is a Gaussian function centered on a target pass rate ( $\tau=0.5$ ). This score estimates how close a candidate's projected difficulty is to the optimal learning frontier.

2. Per-Seed MILP Action Selection

For each seed, a Mixed-Integer Linear Program (MILP) selects the optimal action. The objective function maximizes the frontier score while penalizing the activation of optional skill nodes and unmet coverage targets.

Decision Variables: Binary variables select one projection and one direction per seed.
Constraints: The MILP enforces that exactly one action is chosen per seed and ensures consistency between the selected action and the required/optional skill nodes.
Baseline Mapping: Fixed recipes are represented as restricted masks within this action space (e.g., Few-shot is approximated as diversify + in-depth; Self-Instruct as diversify + in-breadth).

3. Synchronized Environment Rewriting

Once an action is selected, a synthesis model jointly rewrites the entire environment bundle: the instruction, fixtures, oracle solution, test suite, and Docker environment. This ensures the task remains executable and consistent. The prompt explicitly forbids instruction-only edits or hidden test requirements.

4. Gold Verification

Generated environments undergo static checks (schema, path safety, Docker isolation) and gold verification. An environment is accepted into the training pool only if the oracle solution obtains a reward of 1 under the generated tests. This ensures that only runnable, gradeable, and consistent tasks are used for RL training.

Key Contributions

Reframed Synthesis: The paper recasts agent environment synthesis as a reward-grounded action selection process over verified seeds, explicitly defining projections (increase, reduce, diversify) and evolution directions (in-depth, in-breadth).
MILP Formulation: It formulates the selection of projection and direction as a per-task mixed-integer program. This allows for the representation of fixed synthesis recipes as restricted action masks and enables optional soft skill coverage planning.
Synchronized Rewriting & Verification: The method couples the selection decision with the joint rewriting of all environment artifacts, followed by strict gold verification, ensuring that only high-quality, executable tasks enter the RL loop.

Experimental Results

The method was evaluated using Qwen 3.5 models (4B to 35B) trained with Group Relative Policy Optimization (GRPO). All synthesis methods were constrained to export exactly 100 gold-verified environments to ensure a fair comparison of downstream performance and synthesis resource usage.

Performance Gains: On the tb-core benchmark, Envs-FORGE achieved a Pass@1 of 49.2%, a 9.2 percentage point improvement over the Base model (40.0%) and a 2.4 point improvement over the strongest fixed-recipe baseline (Evol-Instruct, 46.8%). On tb-2.0, it reached 29.4% (6.4 points over Base; 2.1 points over Self-Instruct).
SWE-bench Verified: Envs-FORGE achieved 77.1%, outperforming the Base model (73.4%) and all fixed baselines.
Model Scalability: Improvements were consistent across model sizes, with gains ranging from +6.8 points (4B) to +9.2 points (35B) on tb-core.
Resource Efficiency: All methods operated within a comparable synthesis token budget (2.27M–2.88M tokens) and generated a similar number of task directories (194–210). Envs-FORGE required more attempts (291) but utilized shorter attempts on average, demonstrating that the performance gain comes from better task selection rather than increased computational scale.

Significance and Claims

The paper claims that the primary lesson from this work is that environment synthesis for agent RL must decide how each seed should move relative to the current policy before asking an LLM to rewrite it.

By moving away from fixed prompting recipes toward a frontier-aware, reward-grounded selection process, Envs-FORGE ensures that training tasks are neither too easy nor too hard, but rather positioned at the learning frontier where they provide the most effective training signal. The authors emphasize that this approach yields superior downstream performance without increasing the size of the downstream training set or the overall operational scale of the synthesis process. The method provides a structured, inspectable framework for generating high-quality training data for terminal agents.