🤖 AI

Privileged Likelihood Is Not Automatically Value: Three Checks for Token Credit in On-Policy Self-Distillation

This paper argues that privileged token likelihood changes in on-policy self-distillation do not automatically constitute valid outcome credit, demonstrating through formal analysis and empirical experiments that such signals often fail to track better actions or reinforce desired behaviors compared to outcome-only controls.

Original authors: Xuan-Phi Nguyen, Shrey Pandit, Yiran Zhao, Anurag Koul, Zeyu Liu, Shafiq Joty

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xuan-Phi Nguyen, Shrey Pandit, Yiran Zhao, Anurag Koul, Zeyu Liu, Shafiq Joty

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to solve a tricky math puzzle. You show it the problem, and it starts typing out its thoughts, step by step. At the very end, you can easily tell if the robot got the right answer: it's either correct or it's wrong. But what about the steps in the middle? Did the robot make a brilliant move at step three, or did it stumble at step five? In the world of Artificial Intelligence, this is a huge mystery. We know the final result, but we don't know which specific words (or "tokens") the robot wrote were the heroes and which were the villains.

To fix this, researchers have been trying a clever trick called "privileged self-distillation." Think of it like a student taking a test, then immediately getting a reference sheet that explains exactly what they should have done. The student then looks at their own answer, reads the reference sheet, and tries to figure out which words they wrote were good and which were bad. The hope is that by rewarding the "good" words and punishing the "bad" ones, the robot will get smarter faster. But here is the catch: just because a word looks good after you know the answer, doesn't mean it actually helped you get there. It might just be a word that happens to sound nice when you're looking back with hindsight.

This paper from Salesforce AI Research is like a group of skeptical detectives showing up to the classroom to check if this "reference sheet" method is actually working or if it's just a magic trick that fools the teacher. They set up a rigorous experiment to see if the robot is actually learning the right lessons or just memorizing the wrong ones.

The Great "Hindsight" Heist

The researchers decided to put the "privileged self-distillation" method to the ultimate test using a 20-billion-parameter AI model (a very smart robot) and a set of hard math problems called AIME 2025. Their goal was to answer three specific questions that determine if this teaching method is actually useful or just a waste of time.

Question 1: Does the "Good Word" detector actually find the right words?
Imagine a detective trying to find the culprit. If the detective points a finger at a suspect, does that suspect actually have a motive? The researchers checked if the AI's "score" (the number it gives to a word to say how good it was) actually matched up with correct answers. They found that the score was basically guessing. When they looked at thousands of math attempts, the score was only slightly better than a coin flip at telling a correct solution from an incorrect one. In fact, after adjusting for how long the answers were, the score actually liked the wrong answers slightly more than the right ones! It's like a teacher who, after grading a test, gives a gold star to the student who got everything wrong because their handwriting was pretty.

Question 2: Did the robot write its own report card?
This is the sneaky part. In many experiments, the AI generates an answer, and then the same AI (or a version of it) writes a critique of that answer. The researchers worried this was a loop of self-flattery. If you write your own report card, you might be too nice to yourself. To fix this, they tried a "cross-fitting" method: they made the AI write a critique for another AI's answer to the same problem. This breaks the loop. But even with this fix, the scores still didn't reliably predict which answers were correct. The feedback was just as confused as before. It turns out that just because you can write a nice story about a solution doesn't mean the solution is actually good.

Question 3: What happens when the robot tries to learn from these scores?
Finally, they asked: even if the scores were weird, what happens when the robot actually tries to use them to improve? They trained the robot using five different versions of this "token credit" method and compared them to a robot that only learned from the final right-or-wrong answer. The results were a massive disappointment for the "token credit" fans. Every single robot that tried to use the fancy token scores ended up performing worse than the robot that just stuck to the simple right-or-wrong feedback.

The simple robot got about 64.2% of the answers right.
The fancy robots using token scores only got between 24.2% and 33.9% right.

It's as if the robot that tried to analyze every single step of its thinking got confused and forgot how to solve the problem entirely, while the robot that just looked at the final result kept getting better.

The Verdict

The paper concludes that a "token likelihood score"—a number that says a word is more likely to be right because of a reference sheet—is not automatically valuable. Just because a word becomes more probable after you see the answer doesn't mean that word was the key to the solution.

The researchers found that:

The scores didn't track success; they were basically random noise.
Whether the feedback came from the same answer or a different one didn't magically fix the problem.
Using these scores to train the model actually made it worse, pushing it away from the correct answers.

In short, the paper argues that we cannot simply assume that "what looks good in hindsight" is "what is actually helpful." Before we start giving AI models credit for every word they type, we need to check three things: does the score actually match the outcome? Is the feedback fair and independent? And does the training actually make the model smarter? In this specific experiment, the answer to all three was a resounding "no." The fancy method of giving credit to individual words failed, and the old-school method of just checking the final answer was the clear winner.

Technical Summary: Privileged Likelihood Is Not Automatically Value

Problem Statement

In on-policy self-distillation for large language models (LLMs), researchers often attempt to assign token-level credit by using "privileged" information available only during training (e.g., reference solutions, hindsight critiques, or environment feedback) to rescore a model's own rollouts. The central hypothesis is that if a token becomes more likely when conditioned on privileged information, it represents a "correct" or "helpful" step.

However, this paper argues that a change in token likelihood (privileged likelihood) is not automatically equivalent to outcome credit (action value). The authors identify a critical gap: a token likelihood score can be structurally well-formed and active without actually tracking task success or inducing helpful parameter updates. The paper seeks to formally separate three distinct questions that are often conflated in existing self-distillation methods:

Does the score track task success?
Are rollouts compared under a consistent scoring rule?
What behavior does the training loss reinforce?

Methodology and Framework

The authors propose a formal framework to analyze privileged self-distillation, distinguishing between token likelihood scores and action values.

Formal Definitions

Privileged Likelihood Score ( $d$ ): Measures how a scorer's probability for a token $v$ changes when the text context changes from a baseline $c^-$ to a privileged context $c^+$ (e.g., a critique).
$d(s, v) = \log q(v | s, c^+) - \log q(v | s, c^-)$
Action Value ( $Q^\pi_U$ ): The expected terminal utility if the next token is set to $v$ and generation continues under policy $\pi$ .

The paper establishes that $d$ and $Q^\pi_U$ are not inherently equivalent. For $d$ to represent $Q^\pi_U$ , specific conditions (such as positive-affine equivalence) must be met, which are not guaranteed by free-form text feedback.

The Three Checks

The methodology revolves around three specific diagnostic questions:

Q1: Does teacher preference track task success?
The paper investigates whether tokens favored by the privileged context actually lead to better eventual outcomes. It distinguishes between ranking correctness and identifying the specific value of a token action. The authors prove that likelihood ratios only represent principled credit if the conditioning variables are coherent outcome conditionals derived from a joint distribution, which free-form critiques often are not.
Q2: Are rollouts compared under the same scoring rule?
The authors highlight a "self-dependence" issue: if a rollout $Y$ generates its own feedback $C(Y)$ , and $Y$ is then scored by $C(Y)$ , the score measures both solution quality and how well $Y$ matches its own description. To address this, they introduce cross-fitting, where feedback is generated from a different rollout of the same problem (a donor) to score the target rollout. This removes direct self-dependence but does not guarantee the score is useful.
Q3: What behavior does the training loss reinforce?
Even if a score tracks success, the training loss determines how that score updates the model. The paper analyzes how different loss functions (e.g., Forward KL, Reverse KL, bounded rewards) transform token scores into parameter gradients. They demonstrate that scalar invariants (like preserving the sign or total weight of a score) do not guarantee a helpful parameter update direction, especially when the model is confident in a token.

Experimental Setup

Model: gpt-oss-20b (20 billion parameters).
Dataset: AIME 2025 (Mathematical Association of America competition problems).
Feedback Generation: An external LLM judge reads the problem, reference solution, verifier result, and full trace to write a "helpful" explanation ( $C^+$ ) and a "plausible but incorrect" counterpart ( $C^-$ ).
Scoring: A frozen or earlier checkpoint of the model calculates the token likelihood differences based on these feedback texts.
Training Variants: The study compares an outcome-only control (using only final correctness labels) against five variants that incorporate token scores (additive variants with entropy weighting, source clipping, and KL-regularized variants).

Key Results

The experiments provide concrete evidence that privileged likelihood scores often fail to act as useful outcome credit in this setting.

Q1 Results (Tracking Success):
- The implemented additive token score performed near chance in distinguishing correct from incorrect trajectories.
- AUC (Area Under Curve): 0.505 (95% CI: [0.483, 0.524]), which is statistically indistinguishable from random guessing (0.5).
- After adjusting for response length, the score slightly favored incorrect trajectories (AUC = 0.474).
- One-sided teacher residuals were negative for both correct and incorrect rollouts, indicating the teacher down-scores both, merely less so for correct ones.
Q2 Results (Feedback Construction):
- Removing self-dependence via cross-fitting (held-out feedback) did not improve the score's ability to track outcomes. The AUC for held-out masked feedback remained near chance (0.512 raw, 0.485 length-adjusted).
- Scores derived from "own-rollout" feedback appeared stronger (AUC ~0.588) but were shown to be self-dependent, measuring the match between the solution and its own critique rather than objective utility.
Q3 Results (Training Behavior):
- Performance Degradation: All five token-score training variants significantly underperformed the outcome-only control.
- Accuracy Gap: The outcome-only control achieved 64.2% accuracy (avg@4). The token-score variants achieved accuracies ranging from 24.2% to 33.9%, representing a performance drop of 30.4 to 40.0 percentage points compared to the control.
- Mechanism: The token-score variants moved toward higher token entropy and higher KL divergence from the reference policy, whereas the outcome-only control improved accuracy without these shifts. The token scores appeared to reinforce high-entropy, irrelevant tokens rather than correct reasoning steps.

Key Contributions

Formal Distinction: The paper provides a formal framework separating token likelihood scores from action values, proving that likelihood changes do not automatically imply outcome credit.
Three-Check Framework: It organizes the evaluation of privileged self-distillation into three distinct checks (tracking success, feedback construction, and loss behavior), demonstrating that passing one does not imply passing the others.
Theoretical Proofs:
- Proves that the conditional mutual information term in forward-KL decomposition is constant with respect to the student under fixed targets, meaning it is not a task gradient.
- Demonstrates that preserving the scalar sign or total weight of token scores does not preserve the expected parameter gradient.
Empirical Demonstration: Using matched experiments on AIME 2025, the paper shows that a structurally sound privileged likelihood signal can be active yet provide near-zero or negative utility, leading to significant performance degradation compared to outcome-only training.

Significance and Claims

The paper claims that privileged likelihood is not automatically value. The authors emphasize that treating a likelihood signal as credit requires rigorous validation of:

Whether the score actually correlates with task success.
Whether the feedback construction avoids self-dependence loops.
Whether the specific training loss reinforces the desired behavior.

The authors state modestly that these results do not prove that all self-distillation methods fail, but rather that the specific mechanism of using privileged likelihood as a direct credit signal is fragile and often ineffective without passing the three checks. The findings suggest that successful methods must explicitly validate the alignment between the likelihood signal and the task outcome, rather than assuming the teacher's preference under hindsight is a proxy for contribution.

The paper concludes that while the formal distinctions apply generally, the numerical results are specific to the gpt-oss-20b and AIME 2025 setup. However, the identified failure modes (self-dependence, loss-induced gradient misalignment, and lack of outcome tracking) serve as critical diagnostics for future research in on-policy self-distillation.