🤖 machine learning

CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

This paper introduces CalibForge, an autonomous system that synthesizes effective terminal training tasks by using adversarial solver calibration to identify a "learnable zone" where tasks are solvable yet challenging, resulting in significant performance gains across multiple agent benchmarks compared to traditional data curation methods.

Original authors: Fanzhe Meng, Guoxin Chen, Jiale Zhao, Shuang Sun, Zhiyu Lin, Wayne Xin Zhao, Ruihua Song, Ji-Rong Wen, Kai Jia

Published 2026-08-07

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Fanzhe Meng, Guoxin Chen, Jiale Zhao, Shuang Sun, Zhiyu Lin, Wayne Xin Zhao, Ruihua Song, Ji-Rong Wen, Kai Jia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to fix things in a digital workshop. You give it a toolbox and a list of jobs, but there's a catch: if the job is too easy, the robot gets bored and learns nothing; if it's too hard, the robot gives up immediately and learns nothing either. The sweet spot for learning is a "Goldilocks" challenge—something the robot can solve if it really thinks hard, but will fail at if it's inattentive or confused. This is the core puzzle of training AI agents: how do you create a massive library of tasks that are perfectly tuned to this learning zone?

For a long time, researchers have tried to build these task libraries by just making sure the tasks can be done (they are "executable") and that they have a clear right or wrong answer. But just because a task can be solved doesn't mean it's a good teacher. A task might be so simple that any robot solves it instantly, or so broken that no robot can ever solve it. The big question is: how do we find the tasks that are just hard enough to make the robot smarter?

This is where a new system called CalibForge steps in. Think of CalibForge not as a task generator, but as a mischievous game master who plays "adversarial" games with the robot. Instead of just checking if a task works, CalibForge hires a team of different "solver" robots (some very smart, some a bit dumber) to try the task. If the smart robot solves it but the dumb one fails, the task is perfect—it's in the learning zone! If everyone solves it, the task is too easy, so CalibForge makes it harder. If everyone fails, the task is broken, so CalibForge fixes it. By constantly tweaking the tasks based on how these different robots react, CalibForge builds a massive library of 5,431 perfectly calibrated challenges. When new robots are trained on this library, they become significantly better at solving real-world computer problems, proving that the "Goldilocks" zone is the secret sauce for teaching AI.

The Problem: Too Easy, Too Hard, or Just Right?

In the world of Artificial Intelligence, specifically for "terminal agents" (robots that type commands into a computer terminal to fix code or manage servers), researchers have been building huge libraries of practice problems. The goal is to train these agents to handle complex, real-world engineering tasks. However, there's a flaw in how these problems are usually made.

Most systems just check two things:

Can it be done? (Is the computer environment set up correctly?)
Is there a right answer? (Does the task have a clear pass/fail test?)

But this is like giving a student a math test where every question is either "What is 2+2?" (too easy) or "Solve this equation that hasn't been invented yet" (impossible). Neither helps the student learn. The paper argues that we need tasks that are solver-relative learnable. This means a task should be solvable by a capable agent, but not solvable by a weaker one. It needs to sit right on the edge of the agent's ability, forcing it to stretch its brain to succeed.

The Solution: The Adversarial Game Master

The authors created CalibForge, an autonomous system that acts like a relentless game master. It doesn't just write tasks; it calibrates them using a process called Adversarial Solver Calibration.

Here is how the magic happens, step-by-step:

The Clue: CalibForge starts with a vague idea, like "fix a broken database" or "recover lost data."
The Draft: An "Author Agent" (a smart AI) writes a full task, including instructions, a virtual computer environment, and a test to see if the job is done.
The Stress Test: Before the task is kept, CalibForge sends it to a panel of "Solver Agents" to try and solve it.
The Calibration Loop: This is the secret sauce. CalibForge uses two specific strategies to decide if a task is good enough:
- Multi-Solver Calibration: It sends the task to a group of different AI models. If everyone solves it, the task is too easy, so CalibForge makes it harder. If no one solves it, the task is broken, so CalibForge fixes it. The task is only kept if there is disagreement: at least one solver succeeds while at least one fails. This proves the task is in the "learning zone."
- Contrastive Calibration: It sends the task to a "Strong Solver" (a very smart AI) and a "Weak Solver" (a less capable AI). The task is only kept if the Strong Solver passes and the Weak Solver fails. This ensures the task is hard enough to challenge the smart one, but not so hard that it's impossible.

If a task doesn't meet these strict criteria, CalibForge doesn't just throw it away. It looks at why the solvers failed or succeeded, then goes back and rewrites the instructions, the environment, or the tests. It repeats this loop up to 50 times until the task is perfectly calibrated.

The Results: A Library of Perfect Challenges

Using this method, CalibForge built a dataset of 5,431 highly calibrated terminal tasks. The researchers then trained two different AI models (a 30-billion parameter model and a 35-billion parameter model) on this data.

The results were impressive. The models trained on CalibForge's tasks significantly outperformed models trained on other existing datasets:

On Terminal-Bench 2.0 (a standard test for terminal agents), the models scored 32.58% and 47.57%, beating the previous best results by a wide margin.
The improvements weren't just on the training data. When tested on completely different, real-world software engineering challenges (SWE-bench Pro and Doc2Repo), the models showed massive gains, improving by 27.68 and 30.04 percentage points respectively over their base versions.

Why This Matters

The paper suggests that the key to building better AI agents isn't just giving them more data, but giving them the right kind of data. By using the behavior of different solvers to act as a feedback loop, CalibForge ensures that every task in the library is a genuine learning opportunity.

The authors explicitly show that simply adding more solver feedback or using a single solver to check tasks isn't enough. The "Goldilocks" effect—where a task is hard for some but easy for others—is what drives the learning. This approach turns the task creation process from a static "write and hope" method into a dynamic, self-correcting system that actively hunts for the perfect level of difficulty.

In short, CalibForge proves that if you want to train a super-smart robot, you shouldn't just give it a pile of homework. You should give it a pile of homework that is perfectly tuned to make it think, struggle, and eventually succeed.

Technical Summary: CalibForge: Adversarial Solver Calibration for Scaling Learnable Terminal Tasks

Problem Statement

Training terminal agents requires executable and verifiable tasks that are not merely solvable, but appropriately challenging for learning. While existing synthesis systems have made progress in constructing terminal tasks at scale, executable validity alone does not reveal whether a task is trivial, effectively unsolvable, or situated within a "learnable zone" relative to a specific solver setting. Current methods often rely on authoring and validation alone or single-solver feedback, which fails to capture the nuanced behavioral evidence required to guide task construction toward effective training data. The core challenge is determining how solver behavior (successes, failures, and trajectories) can be systematically used to revise candidate tasks to ensure they are neither too easy nor too hard for the target learning regime.

Methodology: CalibForge

The authors present CalibForge, an autonomous terminal-task synthesis system that treats task construction as a constrained adversarial author–solver loop. The system uses verified solver behavior as construction-time feedback to revise candidate tasks through adversarial solver calibration.

1. Task Authoring and Validation

Starting from a "clue," an authoring agent (DeepSeek-V4-Pro) conducts web research to identify concrete engineering problems (e.g., version-specific bugs, dependency conflicts). It constructs a candidate task $\tau$ comprising:

Instruction: The task description.
Environment: Defined by a Dockerfile and initial files.
Verification: Tests to validate the solution.

Before entering the calibration loop, candidates undergo two-stage validation:

Structural Validation: Checks for build-context files, environment initialization, and ensures all tests fail in the initial state.
Self-Solving: The authoring agent attempts to solve the task in an isolated sandbox to ensure the solution is executable and consistent with the instruction and tests.

2. Adversarial Solver Calibration

Valid candidates enter a multi-round calibration loop governed by a specification $\gamma$ . In each round, solvers attempt the task, and the authoring agent uses the outcomes and full interaction trajectories to revise the task. The loop terminates when a retention criterion $C_\gamma$ is met or after a maximum number of rounds ( $R_{max}=50$ ). CalibForge employs two distinct calibration strategies:

Multi-Solver Calibration: Targets disagreement within a heterogeneous solver pool (e.g., DeepSeek-V4-Flash, GLM-5, Kimi K2.5).
- Criterion: Retain the task if at least one solver succeeds and at least one fails ( $0 < \sum y_i < K$ ).
- Goal: Capture a broader range of solution paths and failure modes, reducing reliance on any single solver's capabilities.
Contrastive Solver Calibration: Targets a designated strong-pass/weak-fail relation.
- Criterion: Retain the task if a designated stronger solver (DeepSeek-V4-Pro) succeeds and a designated weaker solver (DeepSeek-V4-Flash) fails ( $y_s=1 \land y_w=0$ ).
- Goal: Operationalize a solver-relative learnable zone where the task is demonstrably solvable but not trivially so under the specified settings.

If the criterion is not met, the authoring agent uses feedback summaries (pass/fail, step count, self-assessment, failure diagnosis) and full trajectories to revise the task (e.g., removing procedural hints, clarifying ambiguous requirements, or generalizing overly prescriptive verifiers) before re-probing.

Key Contributions

Environment-Level Behavioral Calibration: The paper formulates a construction principle where task learnability is treated as solver-relative. It uses verified solver behavior not just for evaluation, but to determine retention and guide multi-round revision.
CalibForge System: An autonomous synthesis system implementing the above principle via a constrained adversarial loop. It introduces Multi-Solver and Contrastive Solver calibration strategies to target cross-solver disagreement and strong-pass/weak-fail relations, respectively.
Calibrated Dataset: The construction of 5,431 calibrated terminal tasks (1,263 via multi-solver, 4,168 via contrastive calibration). These tasks span 16 domain categories, with a focus on software engineering, system administration, and scientific computing, offering broader domain coverage than prior datasets.

Experimental Results

The authors fine-tuned two Qwen backbones (Qwen3-30B-A3B-Instruct and Qwen3.5-35B-A3B) on trajectories distilled from the CalibForge dataset and evaluated them against baselines (Endless Terminals, SETA-Env, CLI-Gym, TermiGen, TerminalTraj) using a matched distillation and training protocol.

Performance on Benchmarks

Terminal-Bench 2.0:
- CalibForge-30B-A3B achieved 32.58% accuracy (vs. 26.22% for the best baseline, TerminalTraj).
- CalibForge-35B-A3B achieved 47.57% accuracy (vs. 40.82% for TerminalTraj).
- Improvements over base models reached 24.71 and 8.47 percentage points, respectively.
Out-of-Distribution (OOD) Transfer:
- SWE-bench Pro: CalibForge-35B-A3B achieved 44.32% (Resolved Rate), an improvement of 3.03 points over its base model and the highest among evaluated data sources.
- Doc2Repo: CalibForge-35B-A3B achieved 48.77% (Pass Rate), an improvement of 3.85 points over its base model.

Ablation Studies

Solver Feedback vs. Calibration: "No Solver" (authoring + validation only) yielded 22.47% accuracy. "Single Solver" feedback improved this to 24.34%. However, Multi-Solver and Contrastive Solver calibration achieved 29.21% and 31.09%, respectively. This indicates that the gains are driven by the calibration strategy (locating difficulty relative to solver settings) rather than simply adding solver feedback or increasing trajectory volume.
Revision Efficacy: In contrastive calibration, only 19% of candidates initially satisfied the target strong-pass/weak-fail relation after structural validation. Through feedback-driven revision and re-probing, 96% of candidates were eventually retained, demonstrating that many initial mismatches are correctable rather than intrinsic defects.

Significance and Claims

The paper claims that solver-relative learnability is a practical and effective target for constructing agent training data. By treating task construction as an adversarial loop where solver behavior guides revision, CalibForge produces tasks that are demonstrably solvable yet challenging enough to drive learning.

The authors emphasize that:

Executable validity is insufficient: A task can be valid but trivial or unsolvable for the target agent. Calibration makes the "learnable zone" observable and actionable.
Calibration drives transfer: The improvements on OOD benchmarks (SWE-bench Pro, Doc2Repo) suggest that the capabilities learned from solver-calibrated terminal tasks transfer effectively to complex, long-horizon software engineering and repository generation tasks.
Diversity is multidimensional: The CalibForge dataset offers diversity not just in domain coverage but in capability tags, environment complexity, and verification logic, avoiding the narrow domain concentrations found in prior datasets.

The results support the conclusion that adversarial solver calibration yields more effective and transferable supervision than authoring and validation alone or single-solver feedback, providing a scalable pathway for constructing high-quality terminal-agent training data.