Technical Summary: TTRL-CoCoV

Problem Statement

Test-Time Reinforcement Learning (TTRL) has emerged as a label-free paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs) by optimizing policies through self-sampling and internal reward construction, bypassing the need for human annotations or oracle labels. While recent TTRL methods have successfully improved single-pass generation accuracy (Pass@1), the optimization of Pass@k (generation coverage within $k$ samples) remains critically under-explored in label-free settings.

Directly applying standard Pass@k optimization strategies from supervised Reinforcement Learning from Verifiable Rewards (RLVR) to TTRL fails due to three fundamental challenges identified through empirical analysis:

1. Noisy Exploration Signals: Pass@k optimization relies on exploring low-confidence samples. In label-free settings, these samples suffer from severe pseudo-label noise, corrupting the exploration signals and degrading performance over time.
2. Length Variance Collapse: Unlike in supervised settings, Pass@k optimization in TTRL fails to maintain response diversity. Models exhibit a "length collapse" where response standard deviations shrink, indicating premature convergence to rigid, repetitive reasoning templates.
3. Formulation Failure: Standard supervised advantage formulations for Pass@k do not translate effectively to unsupervised environments, leading to a degradation of actual Pass@k performance rather than the intended expansion of the exploratory space.

Methodology: TTRL-CoCoV

The authors propose TTRL-CoCoV (Test-Time Reinforcement Learning with Confidence-Conditioned Verification), a novel framework that employs a single shared-weight model ($\pi_\theta$) acting as both a generator and a verifier. The core innovation is a confidence-conditioned mechanism that dynamically allocates verification resources based on the model's internal consensus confidence ($C_{maj}$) to resolve the identified challenges.

The framework classifies samples into three distinct optimization zones:

1. High-Confidence Exploration Zone ($C_{maj} \ge \tau_{high}$)

• Status: The model exhibits high consensus; pseudo-labels are considered reliable.
• Action: The majority answer is used as the pseudo-label. Verification is activated ($I_{verify}=1$) not for filtering, but to bootstrap the verifier using high-quality data, enabling co-evolution.
• Diversity Mechanism: To prevent the "length collapse" observed in standard TTRL, an exploration-enhancing reward ($R_{div}$) is applied. This reward incentivizes trajectory diversity (deviations from mean response length) for correctly labeled samples, ensuring the model continues to explore varied reasoning strategies even on mastered problems.

2. Low-Confidence Verification Zone ($C_{maj} < \tau_{low}$)

• Status: The model exhibits low consensus; pseudo-labels from majority voting are highly noisy.
• Action: The framework delegates pseudo-label selection to the verifier. The model generates multiple verification trajectories for candidate answers.
• Filtering: Only candidates with a high Verification Pass Rate (VPR) are admitted to the trusted set. If no candidate passes, the sample is skipped to prevent contaminating the model with erroneous gradients. This resolves the "noisy exploration signal" challenge.

3. Medium-Confidence Transition Zone ($\tau_{low} \le C_{maj} < \tau_{high}$)

• Status: Ambiguous confidence levels where verification offers negligible advantage over majority voting.
• Action: Verification is bypassed ($I_{verify}=0$) to save inference overhead. The majority answer is used directly for generator updates without additional exploration rewards.

Training Objective

The total loss is a weighted sum of the generation policy loss ($L_{first}$) and the verification policy loss ($L_{second}$). The verification loss employs an asymmetric soft reward matrix, penalizing false positives (incorrectly validating a wrong answer) more heavily than false negatives to ensure the verifier acts as a stringent filter.

Two-Stage Annealing Strategy

To further bridge the gap between label-free and supervised performance, the authors introduce a post-training policy annealing phase. After the initial "exploration" phase (TTRL-CoCoV), the diversity reward is removed, and the model undergoes a "exploitation" phase focused on single-path Pass@1 maximization. This allows the model to refine the diverse reasoning trajectories accumulated during exploration into highly optimized solutions.

Key Contributions

1. Empirical Diagnosis: The paper provides the first systematic investigation into the failure modes of Pass@k optimization in unsupervised TTRL, identifying noisy signals, diversity collapse, and formulation mismatches as root causes.
2. Confidence-Adaptive Framework: TTRL-CoCoV introduces a mechanism where the generator and verifier co-evolve. High-confidence samples train the verifier, which in turn filters noise for low-confidence samples, creating a virtuous cycle.
3. Exploration-Enhancing Reward: A tailored reward mechanism ($R_{div}$) is designed to sustain trajectory diversity specifically in high-confidence regions, preventing premature convergence.
4. SOTA Performance: The method achieves state-of-the-art results across multiple benchmarks, reversing the typical degradation of Pass@k in label-free settings.

Experimental Results

Experiments were conducted on six reasoning benchmarks (including MATH500, AIME24/25, AMC, GPQA, and DAPO) using Qwen3-4B and Qwen3-8B backbones.

• Pass@1 Improvements: TTRL-CoCoV achieved average absolute gains of +9.8% over standard TTRL on Qwen3-4B and +6.2% on Qwen3-8B.
• Pass@16 Improvements: The method yielded massive gains in generation coverage, with average absolute improvements of +18.7% (Qwen3-4B) and +13.0% (Qwen3-8B) over TTRL.
• Surpassing Supervised Baselines: Through the two-stage annealing strategy, the fully unsupervised TTRL-CoCoV achieved Pass@1 improvements of up to +5.0% over fully supervised GRPO baselines on specific complex reasoning benchmarks (e.g., AIME25, GPQA), effectively breaking the performance ceiling established by supervised methods.
• Stability: The framework maintained stable reward accuracy (>0.8) throughout training, avoiding the late-stage pseudo-label collapse seen in competing methods.

Significance and Claims

The paper claims that TTRL-CoCoV fundamentally addresses the "verification-generation gap" in unsupervised learning. By demonstrating that verification capability generally leads generation capability, the framework leverages the model's own ability to verify its outputs to refine training signals.

The authors assert that their approach successfully bridges the gap between label-free and label-supervised learning. Notably, they claim that by first expanding the exploratory space (Pass@k) and then refining it (annealing), the method can not only match but surpass the performance upper bounds of fully supervised baselines on challenging reasoning tasks, all without requiring any ground-truth labels during the training process.