🤖 AI

Escaping the Verifier: Learning to Reason via Demonstrations

The paper introduces RARO, a novel Inverse Reinforcement Learning framework that trains Large Language Models to reason effectively using only expert demonstrations by employing an adversarial game between a policy and a relativistic critic, thereby achieving strong performance on reasoning tasks without relying on task-specific verifiers.

Original authors: Locke Cai, Max Ryabinin, Ivan Provilkov

Published 2026-06-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Locke Cai, Max Ryabinin, Ivan Provilkov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to solve complex puzzles or write beautiful poetry. Usually, to teach a robot, you need a "referee" who can instantly say, "Yes, that's the right answer," or "No, that's wrong."

In math, the referee is easy: you just check if the numbers add up to the right total.
In poetry or financial analysis, there is no referee. There is no single "correct" poem, and no calculator to tell you if a financial strategy is perfect.

For a long time, scientists thought that without a referee, the robot could only learn by simply memorizing examples (like a student copying a textbook). But the authors of this paper, RARO, say: "Wait, we can teach the robot to think and reason even without a referee, just by showing it examples of experts."

Here is how they did it, using a simple analogy:

The "Taste-Test" Game

Instead of a referee who knows the absolute truth, the authors set up a game between two roles played by the same robot brain:

The Chef (The Policy): This part of the robot tries to cook a meal (solve a problem or write a poem) based on a recipe. It wants to make something so good that it looks like it came from a world-famous Master Chef (the expert data).
The Food Critic (The Relativistic Critic): This part of the robot acts as a judge. But here is the twist: The Critic doesn't just taste one dish and say "Good" or "Bad." Instead, the Critic is given two dishes at the same time: one made by the Master Chef and one made by the Chef (the robot).

The Game Rules:

The Critic gets points if it can correctly guess which plate belongs to the Master Chef and which belongs to the Robot.
The Chef gets points if it can trick the Critic into thinking its dish was made by the Master Chef.

Why This is Special

In the past, if the Robot got really good, the Critic would get confused. If the Robot's answer was perfect, the Critic might just guess randomly, and the Robot wouldn't learn anything new.

The authors added a clever "tie" option. If the Critic can't tell the difference between the Robot's answer and the Expert's answer, it can declare a Tie.

This is like the Critic saying, "These two dishes are so similar I can't tell them apart."
This keeps the game going. The Robot learns that to win, it doesn't just need to be "okay"; it needs to be indistinguishable from the expert.

What They Discovered

The researchers tested this "Taste-Test Game" on three different types of challenges:

The Math Puzzle (Countdown): A game where you have to combine four numbers to get 24.
- Result: The Robot learned to "think" through many combinations, checking its work, and backtracking when it was wrong. It got almost as good as robots trained with a perfect math referee, and much better than robots that just memorized answers.
Harder Math (DeepMath): Complex problems where checking the answer is just as hard as solving it.
- Result: As the Robot got bigger and smarter, it kept getting better at reasoning, showing that this method scales up well.
Writing Poetry: A task with no "right" answer.
- Result: This was the big test. The Robot learned to write poems that were significantly more creative and aligned with the prompt than robots that just memorized examples. It learned to plan its poem (e.g., "I need to focus on sensory details") before writing it.

The Bottom Line

The paper shows that you don't need a perfect "answer key" to teach a robot how to reason. You just need a collection of examples from experts and a way to let the robot play a game of "Spot the Difference" against those experts.

By doing this, the robot learns to generate its own internal "referee" that helps it check its work, correct its mistakes, and come up with high-quality solutions for tasks where no one knows the exact right answer in advance. It turns the robot from a simple copycat into a thoughtful problem-solver.

Technical Summary: Escaping the Verifier: Learning to Reason via Demonstrations

Problem Statement

Training Large Language Models (LLMs) to reason has largely relied on Reinforcement Learning with Verifiable Rewards (RLVR), where models are optimized against ground-truth verifiers (e.g., in mathematics or competitive programming). However, many high-impact reasoning tasks—such as analytical writing, open-ended research, and financial analysis—lack such verifiers due to ambiguous criteria or the wide variance of acceptable answers. While expert demonstrations for these tasks are abundant, they remain under-utilized for reasoning-focused training. Standard Supervised Fine-Tuning (SFT) on these demonstrations often fails to induce the self-correction and search behaviors seen in RLVR, and methods like Reinforcement Learning from Human Feedback (RLHF) are prohibitively expensive due to the need for preference data collection.

Methodology: RARO

The authors propose RARO (Relativistic Adversarial Reasoning Optimization), a framework that learns strong reasoning capabilities from expert demonstrations alone, without task-specific verifiers or human preferences. The method is grounded in Inverse Reinforcement Learning (IRL), framing the problem as an adversarial game between a policy and a critic.

Core Components

Adversarial Setup:
- Policy ( $\pi_\theta$ ): Generates answers (with Chain-of-Thought reasoning) conditioned on a question.
- Relativistic Critic ( $c_\phi$ ): A model that takes a question and a pair of answers (one expert, one policy) and predicts which is better, or declares a tie.
- Objective: The policy is rewarded for deceiving the critic (i.e., having its answer classified as "expert"), while the critic is rewarded for correctly identifying the source (expert vs. policy).
Relativistic Critic & Tie Option:
- Unlike standard binary classification, the critic performs a pairwise comparison. This avoids the "degeneracy" problem where an optimal policy makes the classification task impossible for the critic (random guessing).
- Crucially, the critic is allowed to declare a tie. This provides stable, non-zero reward signals even when the policy approaches expert quality, preventing training instability.
Training Algorithm:
- Joint Optimization: Both the policy and the critic are trained jointly using Group Relative Policy Optimization (GRPO).
- Shared Weights: The same LLM backbone is used for both roles to reduce memory usage and promote generalization.
- Replay Buffer: To prevent catastrophic forgetting and mode collapse (where the policy cycles through a fixed set of strategies to fool the critic), the critic is trained on a mix of current rollouts and a replay buffer of past expert and policy answers.
- Stabilization Techniques: The authors incorporate over-length filtering and remove advantage/length normalization to ensure robust learning.

Key Contributions

Novel IRL Perspective: The paper introduces a principled method for training reasoning models using demonstration data only, bypassing the need for verifiers or preference annotations.
Relativistic Adversarial Framework: It proposes a specific adversarial setup with a "relativistic" critic and a tie option, which the authors identify as critical for stabilizing the training dynamics in non-verifiable domains.
Empirical Validation: The method is validated across three distinct domains:
- Countdown (Controlled): A toy math task where verification is easy.
- DeepMath (General Math): A complex reasoning domain where verification is as hard as generation.
- Poetry Writing (Non-verifiable): An open-ended creative task with no objective ground truth.

Experimental Results

The authors evaluate RARO against strong baselines (SFT, Rationalization, Iterative DPO, and RL-Logit) and, where applicable, the upper-bound RLVR.

Countdown (1.5B Model): RARO achieves 54.4% accuracy, significantly outperforming the best verifier-free baseline (SFT at 40.7%) by +13.7%. It nearly matches the performance of RLVR (57.7%).
DeepMath (7B Model): RARO achieves 57.5% accuracy, outperforming the best verifier-free baseline (RL-Logit at 49.3%) by +8.2%. The method exhibits scaling trends similar to RLVR as model size increases.
Poetry Writing (7B Model): In this non-verifiable domain, RARO achieves a 25.0% win rate against expert poems, a massive improvement over the best baseline (SFT at 5.9%), representing a +19.1% gain.
Scaling Behavior: RARO demonstrates effective scaling with both reasoning token budgets (test-time scaling) and model size. Notably, it learns to perform explicit search and self-correction (e.g., backtracking in Countdown), behaviors absent in SFT baselines.

Significance and Claims

The paper claims that RARO effectively elicits strong reasoning performance from expert demonstrations alone, enabling robust reasoning learning even when task-specific verifiers are unavailable. By formulating the problem through Inverse Reinforcement Learning and utilizing a relativistic adversarial setup, the authors demonstrate that it is possible to train reasoning models without the expensive data collection requirements of RLHF or the strict constraints of RLVR.

The results suggest that RARO is a practical pathway for developing reasoning capabilities in real-world domains (like creative writing and complex analysis) where automatic verification is impossible, potentially reducing reliance on costly human preference collection while maintaining the benefits of reinforcement learning, such as test-time scaling and self-correction.