CoVerRL: Breaking the Consensus Trap in Label-Free Reasoning via Generator-Verifier Co-Evolution

1. Problem Statement: The "Consensus Trap"

The paper addresses a critical failure mode in Label-Free Reinforcement Learning (RL) for Large Language Models (LLMs). Current state-of-the-art methods (e.g., TTRL) rely on majority voting to generate pseudo-labels for training without ground truth. The assumption is that the most frequent answer among multiple reasoning trajectories is correct.

However, the authors identify a phenomenon they term the "Consensus Trap":

• Diversity Collapse: As RL training maximizes self-consistency, the model's output diversity decreases, causing predictions to concentrate around a single answer.
• Reinforcement of Errors: If the majority answer is incorrect (a "self-consistent error"), the model confidently reinforces this error. The reward signal (based on consensus) remains high even though the label is wrong.
• Stagnation: This creates a destructive feedback loop where reward accuracy degrades, and the model fails to recover from systematic errors because pure consensus cannot distinguish between "consistent wrong" and "consistent right."

Prior attempts to solve this using self-verification have failed because LLMs cannot reliably verify their own reasoning without specific training, creating a circular dependency (a model cannot verify what it cannot generate correctly).

2. Methodology: CoVerRL Framework

The authors propose CoVerRL (Co-evolving Verifier and generator via Reinforcement Learning), a framework where a single policy model ($\pi_\theta$) alternates between two roles: a Generator ($\pi_{gen}$) and a Verifier ($\pi_{ver}$). The core innovation is bidirectional bootstrapping: the generator creates pseudo-labels for the verifier, and the verifier filters errors to improve the generator's training signal.

Key Components:

1. Pseudo-Label Generation with Filtering:

   • Majority Voting: The generator produces $N$ reasoning paths. The most frequent answer serves as a draft pseudo-label.
   • Self-Consistent Filtering: The model switches to the Verifier role to scrutinize the positive samples (those matching the draft label). If the majority of verification judgments indicate the reasoning is flawed, the sample is discarded. This specifically targets and removes "self-consistent errors" before they corrupt training.

2. Online Dual-Role Co-Evolution:

   • Contrastive Verifier Training: The verifier is trained without ground truth using contrastive pairs. It compares the majority answer (treated as "Correct") against the lowest-frequency answer (treated as "Incorrect"). This creates a balanced dataset ($|V^+| = |V^-|$) to prevent the verifier from collapsing into rejecting all inputs.
   • Self-Correction: When the verifier identifies an error ($v=0$), the model switches back to the Generator role to produce $K$ revised solutions conditioned on the query, the failed attempt, and the verifier's feedback.

3. Answer-Anchored GRPO:

   • The authors adapt Group Relative Policy Optimization (GRPO) for this multi-turn setting.
   • Standard GRPO groups responses by query prefix.
   • Answer-Anchored GRPO groups verification paths by their target answer. Since multiple verification paths in the positive set target the same answer, they are grouped together to compute advantages. This isolates verification variance while maintaining stable generation optimization.

4. Reward Design:

   • A composite reward $r = r_f + r_a$ combines format adherence and accuracy.
   • For the generator, accuracy is measured against the consensus answer.
   • For the verifier, accuracy is measured against the target judgment (Correct/Incorrect) derived from the contrastive setup.

3. Key Contributions

• Identification of the Consensus Trap: The paper empirically demonstrates that majority-voting-based RL leads to diversity collapse and reward degradation when models converge on systematic errors.
• Co-Evolution Framework: CoVerRL is the first framework to unify generation and verification into a single multi-turn RL process where the two capabilities mutually supervise each other without external labels.
• Bidirectional Bootstrapping: It proves that noisy consensus labels can train a verifier, and a trained verifier can filter those same labels to create high-quality training signals for the generator, breaking the circular dependency.
• Answer-Anchored GRPO: A novel adaptation of GRPO that groups verification trajectories by their target answer rather than just the prompt, improving training stability.

4. Experimental Results

The framework was evaluated on Qwen and Llama model families (ranging from 1.7B to 7B parameters) across four benchmarks: MATH500, AMC, AIME24, and GPQA-Diamond.

• Performance Gains: CoVerRL consistently outperforms the TTRL baseline by 4.7% to 5.9% in final accuracy (Acc.@final) on mathematical reasoning benchmarks.
  • Example: On Qwen3-1.7B, CoVerRL achieved 71.9% on MATH500 vs. TTRL's 65.0%.
• Self-Verification Improvement: The verifier capability improves dramatically, rising from ~55% to over 85% accuracy. This confirms that the model genuinely learns to verify, not just generate.
• Reward Accuracy: Unlike TTRL, where reward accuracy drops from ~95% to ~80% due to the consensus trap, CoVerRL maintains reward accuracy above 90% throughout training.
• Generalization: The learned verification capability transfers well to out-of-domain tasks (GPQA), with CoVerRL-enhanced models surpassing much larger models (e.g., Llama-3-70B) on RewardBench.
• Ablation Studies:
  • Removing the verification filter leads to performance drops, confirming the necessity of filtering self-consistent errors.
  • Removing balanced sampling (equal positive/negative verification sets) causes the verifier to collapse (rejecting all answers).
  • Thinking Mode is crucial; models without extended reasoning during verification suffer from "length collapse" and poor accuracy.

5. Significance

CoVerRL represents a significant step forward in Label-Free RL for reasoning tasks. By solving the "Consensus Trap," it enables LLMs to self-evolve their reasoning capabilities without relying on expensive ground-truth data or external verifiers. The framework demonstrates that generation and verification are not mutually exclusive but mutually reinforcing capabilities. This approach offers a scalable path toward creating robust, self-correcting AI agents capable of handling complex reasoning tasks where ground truth is unavailable.