💬 NLP

Likelihood-Based Reward Designs for General LLM Reasoning

This paper systematically demonstrates that using the log-probability of the reference answer as a reward signal is a superior and versatile method for fine-tuning large language models on chain-of-thought reasoning, outperforming binary rewards in verifiable settings and matching supervised fine-tuning in non-verifiable scenarios while avoiding the pitfalls of probability-based alternatives.

Original authors: Ariel Kwiatkowski, Natasha Butt, Ismail Labiad, Julia Kempe, Yann Ollivier

Published 2026-02-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ariel Kwiatkowski, Natasha Butt, Ismail Labiad, Julia Kempe, Yann Ollivier

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart but inexperienced student (a Large Language Model) how to solve complex problems. The student is great at talking, but they need to learn how to "think out loud" (Chain-of-Thought) before giving an answer.

The paper tackles a specific problem: How do you grade this student when you don't have an answer key?

The Old Way: The "Pass/Fail" Quiz

Traditionally, to teach these models, researchers use a method similar to a strict math teacher.

The student thinks out loud.
The student gives an answer.
The teacher checks the answer key.
- Correct? You get a gold star (Reward = 1).
- Wrong? You get a red X (Reward = 0).

The Problem: This works great for math or coding where there is one right answer. But what if you are asking the student to write a long story, a poem, or a complex proof where there is no single "correct" answer? You can't give a simple "Pass/Fail" grade. Also, if the student gets it wrong 99% of the time, they never get a gold star, and they stop learning because the signal is too sparse.

The New Idea: The "Likelihood" Score

The authors propose a different way to grade. Instead of checking if the answer is right, they check how confident the student is in their own answer.

Think of it like this:

The Old Way: "Did you get the answer right? Yes/No."
The New Way: "How likely was it that you would say exactly those words?"

If the student says the exact same words as the reference answer in the training data, the teacher gives them a score based on the log-probability (a mathematical way of saying "how surprised were you that this happened?").

If the student was very confident and said the right words, they get a high score.
If the student was guessing, they get a low score.

The Big Discovery: "Log-Probability" is the Magic Key

The researchers tested many different ways to use this "confidence" score. They found that one specific method—using the log-probability of the reference answer—was the only one that worked everywhere.

Here is the breakdown of their findings using simple analogies:

1. The "Short Answer" Test (Math & Coding)

The Setup: Short problems with a clear right answer (like a math quiz).
The Result: The "Log-Probability" method worked just as well as the old "Pass/Fail" method at getting the right answer.
The Bonus: It also made the student's answers much more "natural" and less jittery. While the old method made the student guess wildly to get a gold star, the new method made the student sound more confident and smooth, even when they were wrong.

2. The "Long Essay" Test (Stories & Proofs)

The Setup: Long, open-ended tasks where there is no single right answer.
The Problem with Other Methods: Some methods tried to use simple "probability" (just the raw chance of being right). But for long essays, the chance of guessing the exact right words is tiny (like winning the lottery). The score became so small it was basically zero, and the student stopped learning.
The Winner: The "Log-Probability" method handled this perfectly. It didn't crash. It performed just as well as if the teacher had simply shown the student the answer and said, "Memorize this."

3. The "Thinking Short" Surprise

One of the most interesting findings was about how long the student thought.

When using the new "Log-Probability" reward, the student initially started thinking less. They shortened their "Chain of Thought" (the thinking process) to just a few words.
Why? The model realized that for the first few steps, a shorter, simpler thought process actually led to a better score than a long, complicated one.
In Math: The student eventually learned to think longer again as they got better.
In Essays: The student stayed short. They basically stopped "thinking out loud" and just gave the answer directly, acting like a standard memorizer. The paper suggests that for long, open-ended tasks, the model might be doing the "thinking" inside its own brain (hidden layers) rather than writing it down, so forcing it to write a long chain of thought actually hurts performance.

The Takeaway

The paper concludes that using log-probability as a reward is a "universal translator" for AI training.

It bridges the gap between verifiable tasks (math) and non-verifiable tasks (writing).
It doesn't require a special "verifier" or answer key.
It allows the AI to learn from any dataset where a reference answer exists, whether it's a short math problem or a long proof.

In short: Instead of asking the AI "Did you get it right?", asking "How sure were you that you said this?" is a simpler, more powerful way to teach AI to reason, covering everything from math puzzles to long essays.

Technical Summary: Likelihood-Based Reward Designs for General LLM Reasoning

Problem Statement

Fine-tuning Large Language Models (LLMs) for reasoning tasks via Reinforcement Learning (RL) typically relies on binary reward functions (0/1 correctness) derived from verifiable ground truths (e.g., mathematical answers). This paradigm faces two primary limitations:

Reward Design Dependency: It requires specific verifiers for each benchmark, which are unavailable for non-verifiable domains like long-form proofs or open-ended generation.
Reward Sparsity: Binary rewards provide sparse signals, potentially hindering efficient learning, especially in complex reasoning chains.

While recent works have proposed intrinsic rewards or self-rewarding mechanisms, these often fail to surpass correctness-based training or suffer from reward hacking and diversity collapse. Furthermore, existing methods utilizing the probability of reference answers (e.g., VeriFree, JEPO, RLPR) have shown mixed results, particularly failing in non-verifiable settings due to the vanishing probabilities of long, exact-match sequences.

Methodology

The authors investigate likelihood-based rewards derived directly from the probability or log-probability of emitting a reference answer (or any prompt continuation present in the data) given a generated Chain-of-Thought (CoT). This approach avoids external verifiers and leverages signals available at scale.

The study systematically compares several reward variants against standard baselines (Supervised Fine-Tuning without CoT, and standard RL with binary rewards):

Base RL: Standard RL using a binary reward ($1$ if answer matches ground truth, $0$ otherwise).
Probability (VeriFree): Reward equals the raw probability $\pi_\theta(a^*|p, z)$ .
Log-prob: Reward equals the log-probability $\log \pi_\theta(a^*|p, z)$ .
Average Log-prob: Log-probability normalized by answer length ( $\frac{1}{|a^*|} \log \pi_\theta(a^*|p, z)$ ).
JEPO: A Jensen-based ELBO loss variant using group-level rewards.
SFT: Standard supervised fine-tuning predicting the answer directly without CoT.

The experiments utilize the RLOO (Reinforce Leave-One-Out) algorithm for advantage estimation. The evaluation spans:

Verifiable Domains: MATH and DeepScaleR benchmarks (short-answer problems).
Non-verifiable Domains: Alpaca and NuminaProof (long-form answers and theorem proofs).
Models: Llama-3.2-3B-Instruct and Qwen-2.5-3B-Instruct.

Key Contributions and Findings

1. Universality of Log-Probability Rewards

The central finding is that log-probability rewards are the only variant that performs robustly across all settings.

In Verifiable Domains: Log-prob rewards achieve success rates comparable to or better than standard binary RL baselines. Crucially, they yield significantly better perplexity than binary RL, aligning the training objective with the pretraining criterion (next-token prediction).
In Non-verifiable Domains: While probability-based rewards (like VeriFree) and standard RL collapse due to the vanishingly small probabilities of generating exact long-form matches, log-prob rewards remain viable. They perform on par with Supervised Fine-Tuning (SFT), whereas other probability-based methods fail to improve upon the base model.

2. The "CoT Collapse" Phenomenon

A significant observation is the behavior of CoT length during training:

Verifiable Settings: Log-prob rewards initially cause a shortening of the CoT, but the length recovers as training progresses.
Non-verifiable Settings: Log-prob rewards lead to a permanent "collapse" of the CoT, reducing the reasoning trace to a minimal length (approx. 10 tokens). In this regime, the log-prob method effectively behaves like SFT.
Mechanism: The authors hypothesize that early in training, shorter CoTs correlate with higher log-probabilities of the correct answer (likely because the base model is not yet adapted to long reasoning traces). This negative correlation drives the RL agent to shorten the CoT. Attempts to mitigate this via KL regularization or length penalties prevented the collapse but resulted in degraded performance, suggesting that for non-verifiable domains, RL with log-prob rewards currently matches SFT performance only by eliminating the CoT.

3. Comparison with Other Methods

Probability vs. Log-Probability: Raw probability rewards (VeriFree) fail in non-verifiable settings because the probability of an exact match for long answers is near zero, providing no learning signal. Log-probability, being a sum over tokens, provides a denser signal that does not vanish as severely.
Perplexity vs. Success Rate: In verifiable domains, binary RL optimizes for success rate but degrades perplexity (the model becomes "confidently wrong" on incorrect paths). Log-prob rewards optimize for both, maintaining a distribution that is less confident on incorrect answers, thus preserving better overall language modeling quality.

Significance

The paper establishes log-probability rewards as a unified training signal capable of bridging verifiable and non-verifiable reasoning tasks without relying on external correctness labels.

It demonstrates that a single criterion (maximizing the log-likelihood of the reference answer given a CoT) can be applied across the full spectrum of answer types, from short math problems to long-form proofs.
It highlights a trade-off: while log-prob rewards improve perplexity and generalizability, they currently do not surpass SFT in non-verifiable domains unless the CoT is preserved (which requires further investigation into training dynamics or compute scaling).
The work suggests that for non-verifiable domains, the "credit assignment problem" in RL may be too severe for long CoTs under current training budgets, and that the benefits of CoT in these settings may require significantly longer training or different initialization strategies (e.g., warm-starting).

The authors conclude that log-likelihood rewards offer a simple, general recipe for post-training reasoning LLMs, valid across diverse answer types, though the mechanism for sustaining long CoTs in non-verifiable settings remains an open challenge.