Self-Evaluation Is Already There: Eliciting Latent Judge Calibration in Base LLMs with Minimal Data
This paper demonstrates that base large language models already possess a latent ability to predict external judge scores, which can be effectively unlocked and refined into a transferable self-evaluation skill through the proposed Self-Evaluation Elicitation (SEE) method using minimal data.
Original authors:XiuYu Zhang, Yi Shan, Junfeng Fang, Zhenkai Liang
Original authors: XiuYu Zhang, Yi Shan, Junfeng Fang, Zhenkai Liang
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine a large language model (LLM) as a talented but slightly shy student who has already read a massive library of books (pre-training). The paper asks: Can this student guess how a strict teacher would grade their essay before the teacher even sees it?
The researchers discovered that the student actually already knows the answer. Even without special training, the model can "feel" how good its own writing is, though its guesses are a bit fuzzy and overconfident.
To fix this, they invented a method called Self-Evaluation Elicitation (SEE). Think of it as a two-step study session that takes very little time:
The Practice Round (RL): The student writes an essay and then immediately tries to grade it. A "teacher" (an external AI judge) also grades it. The student gets points not just for writing a good essay, but for grading themselves accurately.
The Targeted Correction (Distillation): This is the clever part. The researchers take the student's practice essays and say, "You wrote the essay perfectly, so don't change a word of it. But your self-grade was wrong. Let's just erase your grade and write the teacher's correct grade in its place."
By repeating this cycle only 160 times (a tiny amount of data compared to the thousands usually needed), the model learns to predict the teacher's score with high precision.
The Key Takeaway: The paper argues that the ability to judge quality isn't something we need to teach from scratch. It's like a hidden muscle the model already has; we just need to do a few light exercises to "elicit" (bring out) it. Once trained, the model can reliably predict how good its answers are without needing to ask the teacher for help every time.
Problem Large language models (LLMs) are increasingly evaluated by other LLMs acting as judges. A critical open question is whether a model can predict how an external judge will score its own open-ended outputs, particularly in settings where no verifiable ground truth exists. While recent work has trained models to predict scalar correctness on verifiable tasks (e.g., math), it remains unclear if base models possess the latent ability to approximate multi-attribute quality scores from external judges in open-ended scenarios, and whether this ability requires extensive training or merely elicitation.
Methodology The authors propose Self-Evaluation Elicitation (SEE), a lightweight, cyclic procedure designed to surface latent self-evaluation capabilities using minimal data. SEE alternates between two phases:
Calibration-Coupled RL: The model generates a response followed by an inline self-evaluation block containing scores for five attributes (helpfulness, correctness, coherence, complexity, verbosity). An external judge scores the same response. The reward function combines a quality term (based on evaluative attributes) and a nonlinear calibration term (based on the Mean Absolute Error between predicted and actual scores across all five attributes). This phase optimizes the full response using GRPO.
Masked Judge Distillation: Using rollouts collected from the RL phase, the model is fine-tuned via supervised learning. Crucially, the loss is applied only to the self-evaluation tokens, replacing the model's predicted scores with the judge's actual scores while leaving the answer tokens untouched. This phase employs stratified round-robin sampling to ensure coverage across the full spectrum of score bins, preventing the model from learning only mid-range predictions.
Key Contributions
Reframing Self-Evaluation: The paper demonstrates that base LLMs already approximate external judge scores to a significant degree before targeted training, reframing self-evaluation as a problem of elicitation rather than acquisition.
Data Efficiency: SEE achieves superior calibration and quality preservation using only 160 unique examples, roughly 31 times fewer than a standard reinforcement learning baseline (Adapted RLCR).
Robustness and Localization: The elicited self-evaluation is sharply localized within the model's own token distribution (the judge's score frequently appears in the top-5 predicted tokens) and generalizes robustly to held-out judges not seen during training.
Results Evaluated on Qwen3-4B-Base across three open-ended benchmarks (LC AlpacaEval 2.0, Arena-Hard-Auto v2.0, WildBench v2) and HelpSteer2 validation:
Baseline Performance: The untrained base model already predicts judge scores with a calibration score of 0.50–0.70, significantly above random chance.
Improvement: After 15 cycles, SEE improves held-out calibration by 0.25–0.66 Mean Absolute Error (MAE) compared to the base model, outperforming Adapted RLCR which uses substantially more data.
Quality Preservation: Unlike methods that might degrade answer quality to improve calibration, SEE maintains or slightly improves answer quality while sharpening self-prediction.
Generalization: When re-scored by different judges (Claude Sonnet 4.6, Gemini 3.1 Flash-Lite), the SEE model maintains its ranking superiority over baselines, indicating the learned capability is a transferable notion of quality rather than overfitting to a specific judge.
Significance The paper argues that judge-aligned self-evaluation is largely a "readout problem" where post-training serves to surface and enhance pre-existing capabilities rather than install new ones. By isolating the self-evaluation tokens during distillation, SEE demonstrates that specific evaluative behaviors can be refined without disturbing the model's generative distribution. This suggests that expensive, large-scale training runs may be unnecessary for aligning models with judge preferences, offering a more efficient path to reliable self-assessment.