💻 computer science

Learning Reasoning Reward Models from Expert Demonstration via Inverse Reinforcement Learning

This paper proposes an adversarial inverse reinforcement learning (AIRL) framework that learns reusable reasoning rewards directly from expert demonstrations, demonstrating that these learned rewards outperform supervised fine-tuning and outcome-based reinforcement learning by improving model performance, serving as effective inference-time rerankers, and enabling cross-task transfer for diagnosing reasoning failures.

Original authors: Claudio Fanconi, Nicolás Astorga, Mihaela van der Schaar

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Claudio Fanconi, Nicolás Astorga, Mihaela van der Schaar

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart, but inexperienced, apprentice chef how to cook a perfect gourmet meal. You have a stack of recipe cards written by a world-class Master Chef (the "Expert Demonstrations").

Currently, there are two main ways to teach the apprentice:

The "Copycat" Method (Supervised Fine-Tuning): You tell the apprentice, "Just copy exactly what the Master Chef wrote on the card."
- The Problem: If the apprentice gets stuck or tries a new ingredient not on the card, they have no idea what to do. They are just mimicking, not truly understanding why a step is good or bad.
The "Taste-Test" Method (Reinforcement Learning): You let the apprentice cook, and only at the very end, you taste the dish. If it's good, you give a gold star. If it's burnt, you give a thumbs down.
- The Problem: If the dish is burnt, you don't know where it went wrong. Did they burn the garlic in step 1? Did they forget the salt in step 5? The feedback is too late and too vague to fix the specific mistake.

The New Solution: The "Smart Food Critic" (This Paper)

The authors of this paper propose a third way. Instead of just making the apprentice copy the card or waiting until the end to taste, they train a Smart Food Critic (an AI Reward Model) to watch the cooking process in real-time.

Here is how they do it, using the paper's Adversarial Inverse Reinforcement Learning (AIRL) framework:

1. Training the Critic (The "Taste-Test" from the Inside)

The researchers don't give the Critic a rulebook. Instead, they show the Critic two types of cooking videos:

Video A: The Master Chef cooking perfectly.
Video B: The Apprentice cooking (sometimes getting it right, sometimes messing up).

The Critic's job is to figure out: "What makes Video A look like a masterpiece and Video B look like a disaster?"

The clever trick here is that the Critic learns to spot good reasoning steps, not just the final result. It learns that "chopping onions evenly" is a good step, even if the final dish is ruined later because of something else. It learns to say, "Great job on the onions! But wait, you just added sugar to a savory soup—that's a bad move!"

2. The "Granularity" of the Critic

The paper experiments with how detailed the Critic's feedback should be:

Sparse (The "End-of-Shift" Critic): Only gives feedback at the very end. "Good job" or "Bad job." It's stable but doesn't help much during the cooking.
Dense (The "Real-Time" Critic): Gives feedback after every single action. "Good chop," "Bad heat," "Perfect seasoning." This is very helpful for fixing mistakes immediately, but it's harder to train because the Critic might get confused by tiny details.

3. How the Apprentice Learns

Once the Critic is trained, the Apprentice (the AI model) starts cooking again. This time, the Critic watches every step and gives a score.

If the Apprentice takes a step that looks like the Master Chef's logic, the Critic gives a high score.
If the Apprentice takes a weird shortcut or makes a logical error, the Critic gives a low score.

The Apprentice uses these scores to learn how to cook better, without needing a human to taste every single dish.

4. The "Second Life" of the Critic

The best part? The Critic doesn't just help the Apprentice learn; it helps after the learning is done.

The "Reranking" Trick: Imagine the Apprentice is asked to cook 16 different versions of the same dish. Usually, you'd just pick the first one. But now, the Critic tastes all 16 versions and picks the absolute best one.
The Result: The paper shows that using this Critic to pick the best attempt can improve the final result by up to 17.4% compared to just guessing.

5. Diagnosing the "Why"

Because the Critic looks at every step, it can act like a detective. If the final dish is bad, the Critic can point exactly to the moment the Apprentice went off track.

Example: "You were doing great until step 7, where you decided to put the cake in the freezer instead of the oven. That's where you went wrong."

Summary in a Nutshell

This paper teaches AI models to think like experts, not just copy them.

They build a Smart Critic that learns from expert examples to understand the process of reasoning.
This Critic helps the AI learn faster and more accurately than just copying.
It helps the AI pick the best answer from many tries.
It can explain exactly where the AI made a mistake, acting like a personal tutor.

It bridges the gap between "blindly copying a teacher" and "guessing until you get it right," giving the AI a true understanding of how to solve problems step-by-step.

1. Problem Statement

Current methods for improving reasoning in Large Language Models (LLMs) face a fundamental trade-off:

Supervised Fine-Tuning (SFT): Relies on imitating expert reasoning traces. While effective for in-distribution tasks, it is purely imitative and fails to provide guidance when the model deviates from demonstrated trajectories (off-demonstration states).
Outcome-Based Reinforcement Learning (RL): Uses a verifier to reward only the final answer. This requires a well-specified, often task-specific external verifier and lacks step-level supervision, making it difficult to localize errors or guide intermediate reasoning steps.

The paper addresses the gap of learning a step-level reasoning reward directly from expert demonstrations without relying on external verifiers or hand-crafted reward functions. The goal is to create a reward model that satisfies three desiderata:

D1 (Training Signal): Serve as an optimization signal for policy learning on self-generated trajectories.
D2 (Inference Assistance): Act as a reranker to select the best reasoning trace under a fixed sampling budget.
D3 (Error Localization): Identify the specific token or step where a reasoning trajectory first deviates from a correct path.

2. Methodology: Adversarial Inverse Reinforcement Learning (AIRL)

The authors propose an Adversarial Inverse Reinforcement Learning (AIRL) framework adapted for LLMs. Instead of treating expert tokens as direct targets (as in SFT), the method infers a dense reward function from expert traces and optimizes the policy to maximize this reward.

Core Components:

Discriminator (Reward Model, $D_\phi$ ): A token-level classifier trained to distinguish between "positive" and "negative" reasoning traces.
- Positive Set ( $D_{pos}$ ): Expert demonstrations + Policy traces that produce the same final answer as the expert.
- Negative Set ( $D_{neg}$ ): Policy traces with different final answers + Synthetic corruptions of expert traces (e.g., flipped numbers, incorrect medical logic).
- Key Innovation: By including answer-consistent policy traces as positives, the discriminator is forced to learn reasoning quality rather than just source style or answer correctness.
Reward Granularity: The framework evaluates three levels of supervision via a binary mask $m_t$ $m_{t}$ :
1. Sparse: Reward only at the final token ( $t=T$ ).
2. Interval: Reward every $k$ tokens.
3. Dense: Reward at every token.
- For sparse/interval masks, rewards are backfilled (propagated) to unmasked tokens to create a dense signal for policy updates.
Policy Optimization: The policy ( $\pi_\theta$ $π_{θ}$ ) is updated using Group Relative Policy Optimization (GRPO).
- Stabilization: To prevent mode collapse and reward hacking, the authors employ:
  - Replay Buffers: Accumulating positive/negative traces across iterations.
  - Multiple Discriminator Steps: Updating the reward model multiple times ( $N_{disc}$ ) per policy update.
  - Reward Clipping: Clipping dense rewards to a symmetric bound ( $\beta$ ) to prevent numerical instability from unbounded logits.
  - Group Standardization: Normalizing rewards within a batch of sampled outputs to reduce variance.

3. Key Contributions

Adaptation of Adversarial IRL for LLMs: The first work to adapt AIRL for learning dense, process-level reasoning rewards directly from expert demonstrations without external verifiers.
Unified Framework for Training and Inference: Demonstrates that a single learned reward model can effectively serve dual purposes: optimizing the policy during training (D1) and reranking candidates at inference time (D2).
Error Localization Capability: Shows that dense rewards can pinpoint the exact step where reasoning fails (D3), enabling diagnostic capabilities that outcome-based RL lacks.

4. Experimental Results

The method was evaluated on three diverse benchmarks: GSM8K (math), MEDREASON (medical), and MMLU-PRO (scientific QA), using various backbone models (Qwen2.5, Llama3, Qwen3).

Training Signal (D1):
- Learned AIRL rewards generally outperformed SFT. The best variant improved over SFT on MedReason, GSM8K, and MMLU-Pro.
- Granularity Trade-off: Sparse rewards were the most stable and reliable. Dense rewards offered higher resolution but were more prone to instability and mode collapse in adversarial training.
Inference Reranking (D2):
- The learned reward significantly improved Best-of-16 reranking performance.
- Key Stat: Up to 17.4 percentage points improvement over random selection on GSM8K (Qwen2.5-7B with Dense reward).
- Combining reward-guided reranking with majority voting yielded the best results at larger sampling budgets.
Generalization & Localization (D3):
- Transferability: Rewards trained on one task (e.g., GSM8K) or backbone (e.g., Llama3) often transferred positively to others, suggesting reusable reasoning signals.
- Localization: Visualizations (Figure 5) confirmed that the dense reward remained positive for correct intermediate steps but dropped sharply at the first erroneous token, effectively diagnosing the failure point.

5. Significance and Conclusion

This work bridges the gap between pure imitation (SFT) and reward-driven optimization (RL). By learning a reusable, process-level reward model solely from demonstrations, the authors enable:

Better Policy Learning: Moving beyond simple imitation to optimize for reasoning quality.
Test-Time Efficiency: Significant performance gains via reranking without retraining the model.
Interpretability: The ability to diagnose where and why a model fails, which is crucial for safety-critical domains like medicine.

The paper concludes that while balancing reward granularity with optimization stability remains a challenge, AIRL provides a powerful, unified framework for enhancing LLM reasoning capabilities.