💬 NLP

Learning to Self-Verify Makes Language Models Better Reasoners

This paper investigates the capability asymmetry between generation and self-verification in large language models and proposes a multi-task reinforcement learning framework that improves both reasoning and verification performance by training them as complementary objectives.

Original authors: Yuxin Chen, Yu Wang, Yi Zhang, Ziang Ye, Zhengzhou Cai, Yaorui Shi, Qi Gu, Hui Su, Xunliang Cai, Xiang Wang, An Zhang, Tat-Seng Chua

Published 2026-02-10

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuxin Chen, Yu Wang, Yi Zhang, Ziang Ye, Zhengzhou Cai, Yaorui Shi, Qi Gu, Hui Su, Xunliang Cai, Xiang Wang, An Zhang, Tat-Seng Chua

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a student how to solve complex math problems. Usually, we focus on one thing: "Get the right answer." We give them a problem, they work through it, and if they get it right, we give them a gold star. This is how most AI models are trained today.

However, this paper discovers a fascinating "glitch" in how AI learns, and they’ve found a clever way to fix it.

The Problem: The "Smart but Clueless" Student

The researchers noticed a strange phenomenon called asymmetry.

Think of it like this: Imagine a student who is a genius at solving math problems but is completely incapable of checking their own work. They might write a three-page long solution, arrive at the wrong answer, and then confidently say, "Yep, that's correct!"

In AI terms, the model gets better at generating answers (the "doing"), but it doesn't automatically get better at verifying them (the "checking"). Even as the AI gets smarter at math, its ability to spot its own mistakes stays stuck. It’s like a chef who can cook a complex meal but can't tell if it's too salty until someone else tastes it.

The Discovery: The "Editor" Effect

The researchers decided to try something backwards. Instead of training the AI to solve problems, they spent time training it solely to be an editor.

They gave the AI a math problem and a solution (which might be right or wrong) and said: "Don't solve anything. Just tell me if this solution is correct or incorrect."

Here is the magic part: By learning how to be a great "editor," the AI actually became a better "writer."

It’s like a writer who spends a year studying grammar and logic. Even if they aren't writing new stories, their brain becomes so tuned to spotting errors that when they finally sit down to write, their first drafts are much cleaner, more logical, and more accurate.

The Three Big Wins

By teaching the AI to "Self-Verify," the researchers achieved three things:

Higher Accuracy: The "Editor-trained" AI was just as good (and sometimes better) at solving problems as the "Solver-trained" AI.
Efficiency (No More Rambling): Standard AI models often "ramble"—they write long, winding paths of logic, hoping they stumble onto the right answer (kind of like a person talking in circles to hide that they are lost). The Self-Verifying AI became much more direct. Because it knows how to check itself, it doesn't need to wander; it takes the most efficient path to the truth.
Better "Test-Time" Thinking: When the AI is actually working on a hard problem, it can now "pause" and ask itself, "Wait, does that step make sense?" This allows it to catch errors in real-time, much like a professional athlete reviewing game footage to correct their form.

The "Multi-Task" Solution

Finally, the researchers didn't just choose one or the other. They created a training program where the AI alternates between being a "Solver" and an "Editor."

It’s like a training regimen for a professional athlete that alternates between playing the game (Generation) and watching film of the game (Verification). By doing both, the AI becomes a much more complete and reliable "reasoner."

Summary in a Nutshell

Old Way: Train the AI to solve problems $\rightarrow$ It gets better at solving, but remains bad at checking its work.

New Way: Train the AI to check work $\rightarrow$ It becomes a master editor, which accidentally makes it a faster, smarter, and more efficient problem solver.

Technical Summary: Learning to Self-Verify Makes Language Models Better Reasoners

1. Problem Statement

The paper addresses a fundamental capability asymmetry in Large Language Models (LLMs) regarding complex reasoning tasks: the gap between generation (the ability to produce a reasoning path and answer) and self-verification (the ability to judge if a given reasoning path and answer are correct).

Current research and observations suggest that:

Generation $\neq$ Verification: Improving a model's ability to solve problems (generation) does not naturally improve its ability to check its own work (verification).
"Fake" Verification: Many models exhibit "aha moments" where they appear to verify, but these steps often have no impact on the final answer, serving as superficial reasoning rather than true error detection.
Inefficiency: Standard generation-only training often leads to long, redundant, or "fake" reasoning traces that do not effectively utilize self-correction.

2. Methodology

The authors propose a shift in training paradigm, moving from treating verification as a mere auxiliary task to treating it as a primary, independent objective.

A. Self-Verification Training Framework
Instead of optimizing for the correct answer, the model is trained to act as a judge. The pipeline involves:

On-Policy Sampling: The model generates multiple candidate solutions for a query.
Post-Processing: A rule-based verifier labels these as correct/incorrect. The authors then filter, balance (ensuring equal correct/incorrect samples), and diversify the data to create high-quality verification training sets.
Optimization: Using Group Relative Policy Optimization (GRPO), the model is trained to maximize a verification reward ( $r_v$ ), which is based on how accurately its judgment matches the ground truth, rather than the correctness of the solution itself.

B. Multi-Task Reinforcement Learning (RL)
To leverage the benefits of both objectives, the authors propose integrating self-verification into standard generation training through two orthogonal strategies:

Stage-wise Initialization (Verify-Init): The model is first trained solely on self-verification to build a strong "judge" policy, which is then used as the starting point for standard generation training.
Alternating Training (Verify-Alter): The model alternates between generation steps (optimizing for generation reward $r_g$ ) and verification steps (optimizing for $r_v$ ), allowing the two objectives to complement each other throughout the training process.

3. Key Contributions

Discovery of Reverse Asymmetry: The authors prove that while generation training fails to improve verification, learning to self-verify effectively improves generation performance.
Efficiency Gains: They demonstrate that self-verification training leads to significantly shorter and more direct reasoning traces (fewer tokens) without sacrificing accuracy.
New Multi-Task Framework: They provide a decoupled RL framework that outperforms standard generation-only and joint-objective (mixed) training methods.
Test-Time Scaling: They show that strong self-verification enables better "test-time scaling," where the model's own judgments can be used to weight majority voting, improving final accuracy.

4. Results

The experiments were conducted on Qwen2.5 models (1.5B, 3B, and 7B) across several mathematical benchmarks (AIME, AMC, MATH500, etc.):

Performance: Models trained solely for self-verification achieved accuracy comparable to (and sometimes exceeding) those trained for generation.
Token Efficiency: For the 7B model, self-verification training achieved comparable performance using only ~25% of the tokens required by generation-only training.
Error Correction: In "corrupted prefix" tests (where errors were injected into reasoning), self-verification-trained models were significantly better at detecting and correcting mistakes compared to generation-trained models.
Integration Success: The Verify-Alter strategy consistently yielded the highest average accuracy across most benchmarks, proving that decoupled multi-task RL is superior to standard methods.

5. Significance

This work shifts the perspective on LLM reasoning from "learning to solve" to "learning to judge." It suggests that self-correction is a driver of reasoning quality, not just a byproduct of it. By empowering models to recognize their own errors, researchers can develop LLMs that are not only more accurate but also more computationally efficient and capable of scaling their intelligence during inference through reliable self-assessment.