💬 NLP

Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning

This paper introduces SCOPE, a test-time reinforcement learning framework that enhances LLM reasoning by replacing simple majority voting with a dynamic subgroup partitioning and confidence-weighted pseudo-label estimation strategy to mitigate confirmation bias and sparse rewards, achieving significant performance gains on challenging mathematical benchmarks.

Original authors: Weiqin Wang, Yile Wang, Kehao Chen, Hui Huang

Published 2026-04-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Weiqin Wang, Yile Wang, Kehao Chen, Hui Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very smart but slightly confused robot how to solve complex math problems. You don't have a teacher (a human) to check its answers, so you have to rely on the robot to teach itself.

This is the world of Test-Time Reinforcement Learning (TTRL). The current best way to do this is called Majority Voting.

The Old Way: The "Popular Vote" Problem

Imagine you ask the robot to solve a math problem 10 times.

6 times, it gives the answer B.
4 times, it gives the answer D.

The old method says: "Okay, B is the winner because it got the most votes! Let's tell the robot that B is the correct answer and punish it for getting D."

The Problem:
What if B is actually wrong, but the robot just got "stuck" in a loop and kept making the same mistake? Or what if D is the correct answer, but the robot was only 60% sure of it, while it was 99% sure of the wrong answer B?
By blindly following the majority, the robot reinforces its own mistakes. It's like a classroom where the loudest student is always right, even if they are wrong. This is called Confirmation Bias.

Also, the old method only gives a "Yes/No" reward. It's like a teacher saying "Good job" or "Bad job" without explaining why or where you went wrong. This is called Sparse Rewards—it's too vague to help the robot learn deeply.

The New Way: SCOPE (The "Smart Team" Approach)

The authors of this paper propose a new method called SCOPE. Think of it as upgrading the robot's learning process from a chaotic town hall meeting to a structured, high-level strategy session.

SCOPE does two main things to fix the problems above:

1. Listening to Confidence, Not Just Votes (The "Expertise" Analogy)

Instead of just counting how many times an answer appears, SCOPE asks: "How sure was the robot when it gave this answer?"

Imagine the robot is a detective.

Answer B (the popular one) was written with shaky hands and a lot of hesitation. The robot was only 40% sure.
Answer D (the minority one) was written with a steady hand and clear logic. The robot was 95% sure.

SCOPE says: "Even though B won the popularity contest, D is the better answer because the robot was more confident in its reasoning."
It weights the votes by confidence. It's like ignoring the loud, unsure crowd and listening to the quiet, confident expert. This stops the robot from reinforcing its own errors.

2. Breaking into Small Teams (The "Subgroup" Analogy)

The old method treats all 10 answers as one big group. If everyone agrees on the wrong answer, the whole group is punished.

SCOPE breaks the 10 answers into smaller teams (subgroups).

Team 1 might agree on Answer A.
Team 2 might agree on Answer B.
Team 3 might agree on Answer D.

Now, instead of one big "Yes/No," the robot gets multiple, diverse targets.

Team 1 learns to be consistent with Answer A.
Team 2 learns to be consistent with Answer B.
Team 3 learns to be consistent with Answer D.

This is like having a debate club instead of a lecture. By forcing the robot to find the "best answer" within smaller, independent groups, it encourages the robot to explore different ways of thinking rather than just copying the majority. It prevents the robot from getting stuck in a "groupthink" loop.

The Magic Ingredient: The "Balancing Act"

How does the robot know how big these teams should be? Should it have 1 team of 10, or 10 teams of 1?

SCOPE uses a clever trick called Pareto Optimization. Imagine a seesaw.

On one side is Quality (getting the right answer).
On the other side is Diversity (trying new, different answers).

If the teams are too small, the robot gets confused by noise. If they are too big, it gets stuck in groupthink. SCOPE automatically adjusts the team size during training to find the perfect balance where the robot learns the most.

The Results: Why It Matters

When the researchers tested this new method:

It solved significantly more difficult math problems (like the AIME competition) than the old methods.
It helped even smaller, weaker models perform like much larger, smarter ones.
It stopped the robot from getting "stuck" in bad habits.

In a Nutshell

The old method was like asking a crowd, "Who is right?" and blindly following the loudest voice.
SCOPE is like asking, "Who is right, and who is most confident they are right?" and then splitting the crowd into small debate groups to ensure everyone is exploring different ideas.

It turns a chaotic, noisy learning process into a precise, confident, and diverse journey toward mastery.

1. Problem Statement

The paper addresses critical limitations in Test-Time Reinforcement Learning (TTRL), a paradigm designed to improve Large Language Model (LLM) reasoning without relying on ground-truth annotated data. Current TTRL methods (e.g., Zuo et al., 2025) rely heavily on majority voting to generate pseudo-labels for reward signals. The authors identify two fundamental flaws in this approach:

Confirmation Bias: Majority voting treats all votes equally, regardless of the model's internal confidence. If the majority consensus aligns with a plausible but incorrect answer, the model reinforces its own errors.
Sparse Rewards: Binary voting (correct/incorrect based on a single global consensus) fails to provide dense, fine-grained feedback. It cannot distinguish between high-quality reasoning paths that happen to be in the minority and low-quality paths that are frequent.

2. Methodology: SCOPE Framework

The authors propose SCOPE (Subgroup-specific Confidence-Weighted Optimization for Pseudo-label Estimation), a framework that integrates model confidence and dynamic subgroup partitioning to generate more reliable reward signals. The pipeline consists of four key components:

A. Step-wise Confidence Weighting

Instead of simple frequency counting, SCOPE calculates Average Step Confidence for each generated response.

Mechanism: Responses are decomposed into reasoning steps (delimited by newlines). The confidence of each step is the average negative log-probability of its top- $k$ tokens.
Application: The final consensus label is determined by confidence-weighted voting. Responses with higher accumulated step confidence are prioritized over those with higher frequency but lower certainty. This allows the system to recover correct answers that are in the minority but highly confident.

B. Subgroup-Specific Pseudo-Label Estimation

To address reward sparsity, SCOPE avoids assigning a single global consensus label to all samples.

Mechanism: The pool of $|G|$ generated responses is partitioned into $n$ distinct subgroups of size $m$ .
Process: For each subgroup, a local consensus label is derived independently using bootstrap sampling and confidence-weighted voting.
Reward Calculation: Each response receives a reward based on its alignment with its specific subgroup's consensus, rather than a global one. This creates denser, more diverse supervision signals.

C. Automatic Subgroup Size Selection (Pareto Optimization)

The optimal subgroup size ( $m$ ) is not fixed; it is dynamically selected during training to balance two competing objectives:

Reasoning Quality ( $q$ ): Maximizing alignment with the local consensus (accuracy).
Exploration ( $e$ ): Preserving solution diversity to avoid mode collapse (number of unique consensus labels).

Optimization: The method constructs a Pareto front of candidate subgroup sizes. It selects the optimal size $m^*$ that minimizes a trade-off distance to an ideal state (high quality, high diversity), controlled by a parameter $\lambda$ .

D. Training Loop

The framework integrates these components into a Group Relative Policy Optimization (GRPO) loop:

Generate candidate responses.
Calculate step-wise confidence.
Select optimal subgroup size $m^*$ via Pareto optimization.
Partition responses into subgroups and derive local consensus labels.
Compute rewards and update the policy.

3. Key Contributions

Novel Framework (SCOPE): A test-time RL framework that mitigates confirmation bias and reward sparsity by combining step-wise confidence weighting with dynamic subgroup partitioning.
Step-wise Confidence: Introduces a mechanism to recover correct minority answers by prioritizing reasoning certainty over simple frequency, effectively filtering out low-confidence errors.
Pareto-Optimized Subgroups: Proposes an automatic method to balance reward density and estimation accuracy, preventing the model from getting stuck in local optima or over-exploring noise.
Empirical Validation: Extensive experiments demonstrating consistent outperformance over state-of-the-art baselines across various model sizes and benchmarks.

4. Experimental Results

The authors evaluated SCOPE on models ranging from 1.5B to 8B parameters (Qwen2.5, Qwen3, LLaMA3.1) across four benchmarks: AIME 2024, AIME 2025, AMC, and MATH-500.

Overall Performance: SCOPE consistently outperformed baselines, including TTRL (Majority Voting), INTUITOR, RLPR, and EVOL-RL.
Key Metrics:
- Qwen3-8B: Achieved a 13.1% relative improvement on the challenging AIME 2025 benchmark and 8.1% on AMC compared to TTRL.
- Qwen2.5-Math-1.5B: Showed a massive 36.5% relative improvement on AIME 2024, proving the method's efficacy even for smaller models that struggle with self-verification.
- Average Gain: Across all benchmarks and models, SCOPE achieved an average relative improvement of roughly 6.9% to 11.9% over TTRL.
Ablation Studies:
- Removing confidence weighting ("w/o Conf") caused significant performance drops (e.g., -5.00% on Qwen3-8B), confirming that confidence is crucial for filtering errors.
- Removing subgroup partitioning ("w/o Subgroup") led to a sharp decline (e.g., -5.83% on Qwen2.5-1.5B), highlighting the necessity of diverse local consensus for exploration.
Pseudo-Label Stability: SCOPE demonstrated higher Pseudo-label Accuracy (PLA) compared to TTRL, indicating reduced consensus drift and more reliable supervision signals.

5. Significance

This work represents a significant shift in unsupervised RL for LLMs by moving away from coarse-grained, frequency-based consensus toward fine-grained, confidence-aware, and diversity-preserving reward mechanisms.

Reliability: It solves the "confirmation bias" problem where models reinforce their own mistakes, a major hurdle in self-improvement.
Efficiency: By providing denser reward signals, it accelerates convergence and improves sample efficiency.
Scalability: The method is model-agnostic and effective across different parameter scales, making it a viable strategy for enhancing reasoning capabilities in real-world settings where ground-truth labels are unavailable.

The code is released at https://github.com/szu-tera/SCOPE.