🤖 AI

CoRE: Consensus Rewards via Equilibrium for Test-Time Reinforcement Learning

The paper introduces CoRE, a test-time reinforcement learning method that replaces brittle majority voting with graph-based consensus rewards derived from replicator dynamics to provide graded, self-supervised signals that improve accuracy and convergence speed across diverse models and benchmarks.

Original authors: Ambuj Mehrish, Sebastiano Vascon

Published 2026-08-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Ambuj Mehrish, Sebastiano Vascon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where a super-smart robot is trying to learn how to solve a tricky puzzle, but nobody is there to tell it if it got the answer right. This is the daily life of modern AI models when they face new, unlabeled questions. To learn, these models usually need a teacher to say, "Good job!" or "Try again!" based on a correct answer key. But what happens when there is no answer key? This is where a field called Test-Time Reinforcement Learning comes in. It's a clever trick where the AI tries to teach itself by generating many different attempts (called "roll-outs") at the same problem and then looking at its own work to figure out what to believe.

The standard way to do this is like a classroom vote. If the AI generates 64 different answers, it simply counts them up. If 35 say "42" and 29 say "17," the AI assumes "42" is the truth because it has the most votes. It then rewards every attempt that said "42" and punishes the rest. This works well most of the time, but it has a fatal flaw: it treats a lucky guess the same as a brilliant, well-reasoned solution, and it ignores the possibility that the "majority" might be confidently wrong. If a small group of smart, confident attempts says "17" while a larger, confused group shouts "42," the voting system will crush the smart minority. This paper asks: Can we build a smarter way for the AI to listen to itself, one that values how the answers agree, not just how many agree?

The Problem with the Ballot Box

Imagine a group of detectives trying to solve a crime. In the old way (the "Majority Vote" method), they just raise their hands. If 60 detectives point to the butler and 40 point to the gardener, the butler is guilty. But what if the 60 pointing at the butler are all confused, while the 40 pointing at the gardener are actually the only ones who read the clues correctly? The voting system would punish the smart detectives and reward the confused ones, making the whole team dumber over time.

This is exactly what happens to AI models using standard Test-Time Reinforcement Learning. They generate a bunch of answers, count the winners, and assume the winner is right. If the AI makes a systematic mistake that happens to be popular, it reinforces that mistake. It's like a broken democracy where the loudest voice wins, even if they are shouting nonsense.

Enter CoRE: The Detective's Roundtable

The authors of this paper, Ambuj Mehrish and Sebastiano Vascon, propose a new method called CoRE (Consensus Rewards via Equilibrium). Instead of just counting heads, CoRE treats the AI's attempts like a complex web of relationships.

Here is how it works, using a simple analogy:
Imagine the AI's 64 attempts are people sitting in a room.

The Graph: Instead of just shouting answers, these people form a giant web of connections. Two people are connected if they gave the same answer. But the connection isn't just "yes/no." It's weighted by how similar their reasoning was (did they use the same logic?) and how confident they sounded (did they speak with certainty?).
The Equilibrium: The system then runs a mathematical simulation called "replicator dynamics." Think of this as a game of musical chairs where the people who are most supported by their confident, logical neighbors start to stand up and form a tight, strong circle. The "dominant set" is the group that holds the most mutual support.
The Reward: Instead of giving a simple "pass" or "fail" to everyone who voted for the winner, CoRE gives a graded score. If your answer was part of a tight, confident, logical circle, you get a high reward. If you were part of a weak, confused group, you get a low reward. Even if you were in the minority, if your group was the most coherent and confident, CoRE might actually pick you as the winner.

What They Found

The researchers tested this new "CoRE" method against the old "Majority Vote" method across seven different AI models and five different math and science benchmarks. They ran the experiments with three different random seeds (basically, three different starting conditions) to make sure the results were real.

The results were quite promising:

Better Scores: On average, the models using CoRE improved their accuracy by 21.7 points compared to their starting point without any learning. The old voting method only improved by 20.4 points.
Winning the Contested Battles: The real magic happened when the answers were split. In situations where the AI was unsure and the "vote" was close, CoRE beat the voting method by up to 7.5 points. It successfully rescued the "smart minority" that the voting system would have ignored.
Faster Learning: CoRE didn't just get better; it got there faster. It reached the same final accuracy level as the voting method in 54% to 70% fewer steps. This suggests that the graded, nuanced rewards from CoRE give the AI a clearer, more helpful signal to learn from.

The "Recovery Zone"

The paper also explains when this works best. They found a specific "recovery zone." If the AI is so bad that it can't solve the problem at all, or so good that everyone agrees perfectly, CoRE acts just like the old voting method (which is fine). But in the messy middle ground—where there is a small group of correct, confident answers fighting against a larger group of wrong, confident answers—CoRE shines.

The authors showed mathematically that if the correct answers are even slightly more confident than the wrong ones, CoRE can flip the script and pick the right answer, even if it's the minority. It's like a wise judge who realizes that the 40 detectives with the solid alibi are more trustworthy than the 60 detectives who just guessed.

The Bottom Line

This paper suggests that we don't need to throw away the voting system entirely; CoRE actually includes voting as a special case. But by adding a layer of "social intelligence"—looking at how the answers support each other and how confident they are—we can turn a brittle, all-or-nothing vote into a smart, calibrated reward system.

The authors are careful to note that this isn't a magic wand that fixes everything. If the AI is completely lost (the "competence floor"), CoRE can't invent a correct answer out of thin air. But for the tricky problems where the AI is almost right but gets confused by the crowd, CoRE suggests a way to listen to the quiet, confident voice of reason rather than just the loudest shout. It turns a simple headcount into a sophisticated conversation, helping AI models learn faster and smarter from their own mistakes.

Technical Summary: CoRE – Consensus Rewards via Equilibrium for Test-Time Reinforcement Learning

Problem Statement

Test-time Reinforcement Learning (TTRL) aims to improve language model reasoning on unlabeled test data by deriving supervision from the model's own roll-outs. The standard approach, Majority Voting, treats the set of $N$ sampled answers as a ballot box: it identifies the most frequent answer as a pseudo-label and assigns a binary reward (1 or 0) to trajectories matching that label.

This approach suffers from three critical limitations:

Brittle Aggregation: It discards correct answers if they constitute a minority, even if they are internally coherent. Conversely, it rewards incorrect majority clusters, potentially amplifying systematic errors.
Binary Rewards: It assigns identical rewards to a well-supported derivation and a lucky guess, failing to distinguish the quality of agreement.
Uniform Question Weighting: It treats every question equally, regardless of whether the roll-outs form a coherent consensus or a fragmented set of incompatible solutions.

The core challenge is to construct a reward signal from unlabeled roll-outs that captures the relational structure and mutual support among reasoning paths, rather than merely counting votes.

Methodology: CoRE

The authors propose CoRE (Consensus Rewards via Equilibrium), a method that replaces majority voting with a graph-based equilibrium approach. CoRE operates on the same $N$ roll-outs as standard TTRL but processes them differently to generate refined signals.

1. Roll-out Graph Construction

CoRE constructs a weighted graph $G=(V, E, \omega)$ where nodes represent the $N$ roll-outs. The edges are defined by an affinity matrix $A'$ that combines three signals:

Answer Agreement: A hard kernel $K_{ans}$ ensures edges only exist between roll-outs with the same extracted answer.
Reasoning Similarity: A soft kernel $K_{rsn}$ based on the cosine similarity of TF-IDF $n$ -gram vectors of the reasoning text strengthens ties within the same answer class.
Confidence Calibration: Node weights $w_i$ are derived from generation confidence (mean token log-probability). Edges are scaled by the geometric mean of the endpoint confidences ( $\sqrt{w_i w_j}$ ), ensuring that high-confidence trajectories reinforce each other more strongly.

The final affinity matrix $A'$ is block-diagonal, where each block corresponds to a unique answer class.

2. Dominant Set Extraction via Replicator Dynamics

Instead of counting votes, CoRE extracts a Dominant Set (DS) from the graph using Replicator Dynamics (RD), an evolutionary game-theoretic optimization process.

Initialization: The system starts from the barycenter of the simplex, assigning initial mass proportional to the size of each answer class (effectively starting from the majority vote).
Dynamics: The system iteratively updates the mass distribution $x$ based on the quadratic form $x^\top A' x$ . This process converges to a stable point $x^*$ where mass concentrates on the most mutually supporting subset of trajectories.
Outcome: The support of $x^*$ (nodes with mass above a threshold) defines the consensus.

3. Reward Generation

From the equilibrium state $x^*$ , CoRE derives three distinct signals:

Refined Pseudo-label ( $y^*$ ): The answer class with the highest equilibrium mass. This may differ from the plurality vote if a smaller, highly coherent minority emerges as dominant.
Graded Rewards ( $R_i$ ): Each roll-out receives a continuous reward proportional to its affinity to the consensus, rather than a binary 0/1.
Cohesiveness Gate ( $w_q$ ): A question-level scalar measuring the overall coherence of the consensus. This gate scales the loss function in the policy update (GRPO), filtering out questions where no reliable consensus exists.

Key Contributions

1. Methodological Innovation

CoRE is the first test-time RL method to utilize Dominant Set extraction to construct rewards. It transforms the roll-out group from a "ballot box" into a "graph," enabling the recovery of correct minority clusters that are coherent and confident, even when outnumbered. It integrates directly into GRPO without requiring auxiliary models or extra roll-outs.

2. Theoretical Analysis

The paper provides a rigorous theoretical framework:

Generalization: Majority voting is proven to be a special case of CoRE (when reasoning similarity is ignored and confidence is uniform).
Recovery Threshold: A block-value analysis derives the exact threshold at which a correct minority can overturn an incorrect plurality. The threshold depends on the ratio of cluster sizes and their internal coherence.
Confidence Multiplication: The analysis proves that confidence calibration multiplicatively lowers the recovery threshold. Specifically, if correct roll-outs are more confident than incorrect ones, the required coherence for a minority to win decreases exponentially with the confidence gap.

3. Empirical Evidence

Evaluated across 7 backbones (Qwen, DeepSeek, Llama, Mistral) and 5 benchmarks (AMC, AIME, MATH-500, MATH-L4/5, GPQA-Diamond) with 42 model-benchmark configurations:

Performance: CoRE improves accuracy by +21.7 points over the untrained base on average across all 42 model–benchmark cells, outperforming Majority-vote TTRL (+20.4 points). Notably, math-specialized models alone show an average improvement of +25.0 points.
Contested Consensus: CoRE achieves gains of up to +7.5 points over voting specifically in scenarios where agreement is contested (e.g., MATH-L4/L5 subsets).
Sample Efficiency: CoRE reaches the final accuracy plateau of the voting baseline in 54–70% fewer optimization steps.
Ablation: Removing either the graph structure or confidence calibration reduces performance to near-majority-vote levels, confirming that the synergy between structural coherence and confidence is essential.

Significance and Claims

The paper claims that consensus is not merely counting. By modeling the mutual support among reasoning paths, CoRE converts a brittle voting mechanism into a calibrated, graded, self-supervised reward signal.

Mechanism: The method succeeds by recovering supervision from "smaller, coherent, and confident correct groups" that vote counts miss.
Operating Regimes: The gains are concentrated in the "intermediate regime" where a coherent correct minority exists but is not the plurality. In "saturation" regimes (near-unanimous correct answers) or "competence floor" regimes (random guessing), CoRE naturally degrades to or performs similarly to majority voting, avoiding overfitting to noise.
Self-Supervision: CoRE demonstrates that treating roll-outs as a graph rather than independent samples yields a denser training signal, improving both the quality of the pseudo-label and the efficiency of the policy update.

The authors conclude that CoRE represents a substantive component of self-supervised RL, offering a principled way to handle disagreement in test-time adaptation without external supervision.