🤖 AI

Reward-Free Evolving Agents via Pairwise Validator

This paper proposes a cost-effective self-evolving agent framework that replaces expensive scalar reward signals with a frozen LLM-based pairwise validator to guide agent improvement, achieving competitive performance across multiple engines and substrates without requiring labeled data or additional training.

Original authors: Minghao Liu, Yu Wang, Jiayun Wang, Wei Wei

Published 2026-07-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Minghao Liu, Yu Wang, Jiayun Wang, Wei Wei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers don't just follow orders but actually learn to write their own instructions. This is the realm of AI Agents, smart programs that solve complex puzzles by breaking them down into steps, using tools, and thinking through problems. Usually, to make these agents better, humans have to act like strict coaches. Every time the agent tries a new way of thinking, the coach has to grade it with a specific score, like a teacher giving a test grade. But here's the catch: creating those perfect tests and grading keys is incredibly hard, expensive, and requires deep human expertise. It's like trying to teach a robot to cook by writing a 100-page manual on exactly how to chop an onion, only to realize you need a new manual for every single vegetable.

Enter the concept of Self-Evolving Agents. Instead of waiting for a human coach to grade every attempt, these agents try to improve themselves in a loop: they make a small change, test it, and if it looks better, they keep it. The big question scientists have been asking is: Can we skip the expensive human grading entirely? What if the agent could just look at its "old self" and its "new self" and decide which one is better, without needing a perfect score? This paper dives into that exact question, proposing a clever shortcut that replaces the heavy, expensive grading system with a simple, quick comparison.

The Problem: The Expensive "Scorecard"

Think of building a super-smart AI agent like training a dog to do tricks. In the old way, every time the dog tries a new move, a human trainer has to step in, watch closely, and give it a precise score: "That was a 7.5 out of 10." To get a reliable score, the trainer needs a massive library of perfect examples to compare against. For AI, this "scorecard" is called a scalar reward. It's a single number that tells the agent how well it did.

The problem is that making this scorecard is a nightmare. It requires hiring experts, labeling thousands of examples, and it costs a fortune. Sometimes, the task is so weird or complex that humans can't even agree on what a "good" score looks like. It's like trying to grade a painting of a dream: there's no right answer, so giving it a number is impossible.

The Solution: The "Who's Better?" Game

The authors of this paper, Minghao Liu and his team, asked a simple question: What if we stop asking "How good is this?" and start asking "Which one is better?"

Instead of a human (or a complex computer) trying to assign a difficult number to a single attempt, they introduced a Pairwise Validator. Imagine two contestants in a boxing ring: the "Parent" (the agent's current version) and the "Child" (the new, tweaked version). A frozen, pre-trained Large Language Model (LLM) acts as the referee. It doesn't need to know the exact score; it just looks at both and says, "I think the Child is better," or "The Parent wins."

This is much easier for the AI referee. It's like asking a friend, "Do you prefer chocolate or vanilla?" rather than asking them to rate both flavors on a scale of 1 to 100 with perfect precision. The paper shows that this "Who's Better?" game is more stable and requires zero extra training for the referee.

The Two New Playbooks

The team tested this idea in two different ways, creating two new "playbooks" for the agents:

Adaptive Focus: This is the "Smart Coach" approach. The agent still uses a small set of human-graded examples to pick which version to keep as the "parent" for the next round. However, at the moment of deciding whether to accept a new change, it skips the expensive scorecard and just uses the "Who's Better?" referee. It's a hybrid that saves money on the daily decisions but keeps the human safety net for long-term planning.
Soft Elo: This is the "Pure Tournament" approach. Here, the agent throws away the human scorecard entirely. It uses the referee's "Who's Better?" verdicts to run a rating system (similar to how chess players are ranked). If the Child beats the Parent, the Child gets a rating boost. Over time, the agent evolves based purely on these head-to-head battles, never needing a human to assign a number.

The Results: Does It Work?

The team put these methods to the test across a wide variety of challenges, from answering tricky trivia questions and solving math problems to writing code that organizes data. They compared their new "Pairwise" methods against the old "Full-Reward" methods (the expensive, human-scored way).

The findings were surprisingly strong. In most of the tests, the agents using the "Who's Better?" referee performed just as well as, or even better than, the agents using the expensive human scorecards.

For Trivia and Instructions: On tasks like answering questions from a set of documents, the "Soft Elo" method actually helped the agent generalize better, meaning it didn't just memorize the practice questions but learned the actual logic.
For Math: Even on hard math problems, the new methods managed to solve the same number of problems as the old methods, proving that you don't need a perfect score to find a good solution.
For Code: When evolving computer programs, the pairwise gate worked just as well as the full-reward baseline, successfully guiding the code to become more efficient.

The Catch and the Future

The paper is careful to note that this isn't a magic wand for every situation. The method works best when there is still room for the agent to improve. If the agent is already at the top of its game, or if the referee (the LLM) is confused by the task, the system can get stuck. Also, if the "test" is too small (like a math quiz with only 15 questions), the results can look a bit noisy, just like flipping a coin a few times doesn't tell you if it's fair.

However, the core discovery is a game-changer: You don't need a perfect, expensive scorecard to evolve a smart AI. By simply asking a frozen AI referee to compare two versions and pick the winner, we can build self-improving agents that are just as capable as the ones we build with human experts, but without the massive cost and effort of labeling data. It's a shift from "grading every paper" to "just picking the best essay," and it turns out, that's enough to teach a machine how to think.

Technical Summary: Reward-Free Evolving Agents via Pairwise Validator

Problem Statement

The construction of self-evolving agentic loops—systems that iteratively refine an agent's prompt, code, or workflow without retraining the underlying model—faces a significant bottleneck: the design of the per-iteration quality signal (reward). Currently, these loops rely on a scalar reward to accept or reject candidate modifications. Designing a reliable scalar reward is often prohibitively expensive, requiring domain expertise and large sets of clean, labeled examples that can cost as much as the agent's primary task itself. In many scenarios, defining such a reward is difficult or impossible, creating a practical asymmetry where collecting a few examples of "good" vs. "bad" outputs is easy, but collecting enough data to train a robust reward function is not.

Methodology

The authors propose replacing the scalar reward at the accept/reject gate of a self-evolving loop with a pairwise validator. This validator is a frozen Large Language Model (LLM) that, given a parent candidate and a child candidate, returns a binary verdict on which is better. This approach leverages the finding that pairwise judgment is generally more stable and easier for LLMs than absolute scoring, as it avoids the need for strict scale calibration.

The paper introduces two specific configurations of this method, integrated into existing engines (GEPA, ADRS, ShinkaEvolve):

Adaptive Focus (Train-Reward-Free):
- Mechanism: Replaces the scalar reward at the accept/reject gate with the pairwise validator.
- Parent Selection: Retains the engine's existing parent selection mechanism, which relies on validation-set rewards.
- Context: The validator's context ( $E$ ) can be adjusted. The "Adaptive Focus" variant specifically employs a meta-LLM to periodically refresh the evaluation rubric (e.g., shifting focus between accuracy, soundness, or holistic quality) based on recent gate activity.
Soft Elo (Fully Reward-Free):
- Mechanism: Replaces the scalar reward at the gate with the pairwise validator.
- Parent Selection: Completely removes the reliance on validation-set rewards. Instead, it implements an Elo rating system driven by the validator's verdicts.
- Process: Candidates in the pool are assigned ratings. In each iteration, a parent is selected via softmax over ratings. The child is matched against the parent and random archive members. The validator judges these pairs, and Elo ratings are updated based on the outcomes.
- Confidence Weighting: The update step size is modulated by the agent's per-token confidence ( $c_t$ ) from the previous iteration, making the system "Soft" Elo. If per-token probabilities are unavailable, a "Plain Elo" variant with a fixed step size is used.

Key Contributions

Reward-Free Validation Variants: The paper defines two methods to eliminate the need for per-step task rewards. Both are training-free (the validator is a frozen LLM). Adaptive Focus removes the train reward but keeps validation rewards for selection; Soft Elo removes both, relying entirely on pairwise verdicts for both gating and parent selection.
Empirical Superiority: The authors demonstrate that their reward-free methods match or exceed the performance of full-reward baselines across multiple agents (Qwen3, Gemma-4, gpt-oss) and tasks (retrieval, instruction following, math, code evolution).
Generalization and Robustness: The performance gains persist even when the validator model is swapped with a different model family (e.g., using Claude Haiku or Gemma-4 as the validator while the agent remains Qwen3), suggesting the benefits are not artifacts of self-evaluation.
Reduced Validation-to-Test Gap: In settings where validation sets are small (low-data regime), the pairwise gate demonstrates a smaller gap between validation performance and test performance compared to scalar reward baselines, suggesting better generalization.

Results

The evaluation covers ten tasks across three families (retrieval/instruction-following, math, code) using three engines.

Prompt Evolution (GEPA): Across five agent configurations and four NLP tasks, the proposed methods (Adaptive Focus and Soft Elo) matched or outperformed the full-reward GEPA baseline in the majority of cases. Notable improvements included Soft Elo on HoVer (+10.0) and Adaptive Focus on IFBench (+11.6).
Math Tasks: On AIME and LiveBench-Math, the methods achieved parity or significant gains (e.g., +20.0 on AIME for Qwen3-8B thinking) despite the small test set sizes, serving as a robustness check.
Code Evolution (ADRS, ShinkaEvolve): On code tasks, the "Direct + full reward" variant (pairwise gate + full reward selection) won 5 of 10 cells, while "Direct + easy reward" won 6. The pairwise gate maintained parity or superiority where there was room for improvement, though it did not surpass the baseline in saturated regimes (e.g., signal processing).
Cross-Validator Swap: When the validator was replaced with a different model family (Haiku or Gemma-4), the reward-free methods continued to match or exceed the baseline, confirming the method's validity beyond self-paired setups.

Significance and Claims

The paper claims that the pairwise validator serves as a drop-in replacement for per-step reward design. It achieves competitive task accuracy without the labeling cost associated with training a scalar reward model.

The authors emphasize that the substitution works best when:

The baseline has room to improve (non-saturated regimes).
The validator can reliably distinguish between parent and child candidates.

They modestly acknowledge limitations where the method may fail:

Saturation: When the baseline is already near the agent's performance ceiling.
Small-N Test Sets: Where single examples disproportionately affect scores.
Capability Gaps: When the agent fundamentally lacks the ability to solve the task.
Validator Mismatch: When the validator itself struggles with the specific task type, leading to noisy verdicts.

Ultimately, the paper positions the pairwise gate not as a universal replacement for all reward scenarios, but as a highly effective, cost-efficient alternative for the specific bottleneck of designing per-step rewards in self-evolving agent loops.