💬 NLP

Through the Judge's Eyes: Inferred Thinking Traces Improve Reliability of LLM Raters

This paper introduces a human-LLM collaborative framework that uses rejection sampling to infer thinking traces from label-only annotations, which are then utilized to fine-tune open LLM raters and refine guidelines for proprietary models, significantly improving their reliability and agreement with human judgments.

Original authors: Xingjian Zhang, Tianhong Gao, Suliang Jin, Tianhao Wang, Teng Ye, Eytan Adar, Qiaozhu Mei

Published 2026-02-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xingjian Zhang, Tianhong Gao, Suliang Jin, Tianhao Wang, Teng Ye, Eytan Adar, Qiaozhu Mei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of super-smart robots to judge a talent show. You give them a list of rules (a "codebook") and a bunch of performances to rate. But here's the problem: the robots are great at following instructions, but they often get the spirit of the rules wrong. They might give a 5-star rating to a messy performance just because it was loud, while a human judge would give it a 2 because the story didn't make sense.

The robots are missing something crucial: the "why." They see the final score, but they don't see the human judge's internal monologue—the messy, step-by-step thinking that led to that score.

This paper, "Through the Judge's Eyes," is about teaching robots to think like humans by reverse-engineering their thought processes. Here is the story of how they did it, using some simple analogies.

1. The Problem: The "Black Box" of Judgment

Imagine a human judge watching a play. They think: "The lighting was great, but the actor forgot their lines, and the plot was confusing. So, I'll give it a 3."
The robot, however, only sees the final number: 3. It doesn't know why the judge picked 3. Without knowing the reasoning, the robot is just guessing. It's like trying to learn how to bake a cake by only looking at the finished cake, without ever seeing the recipe or the mixing process.

2. The Solution: The "Ghost Writer" (Inferred Thinking Traces)

The researchers realized that while they couldn't ask every human judge to write down their thoughts (it's too slow and expensive), they could use a special kind of AI called a Reasoning Language Model (RLM). Think of this RLM as a "Ghost Writer" who is very good at talking out loud while it thinks.

The Magic Trick: Rejection Sampling
Here is how they got the Ghost Writer to mimic human thinking:

The Setup: They took a story and the human's final score (e.g., "Complexity: 3").
The Guessing Game: They asked the Ghost Writer to write a thinking process and guess the score 16 times.
- Attempt 1: "This story is boring. Score: 1." (Wrong score, throw it away).
- Attempt 2: "The plot is wild but the ending is weak. Score: 3." (Correct score! Keep this thought process).
- Attempt 3: "It's perfect. Score: 5." (Wrong score, throw it away).
The Result: They kept only the thoughts that led to the correct human score.

By doing this thousands of times, they built a massive library of "Inferred Thinking Traces." These aren't the exact thoughts of the original human, but they are high-quality guesses that sound exactly like how a human would reason to get that specific score.

3. Two Ways to Use the "Ghost Thoughts"

Once they had this library of "human-like thinking," they used it in two clever ways to fix the robot judges.

Method A: The "Tutoring" Approach (Fine-Tuning)

For robots that can be reprogrammed (open-source models), they used these thinking traces as a textbook.

Before: The robot was trained on "Story + Score = 3."
After: The robot was trained on "Story + Here is the step-by-step reasoning + Score = 3."
The Analogy: It's like a student who used to just memorize the answer key. Now, they are being taught the math behind the answer. They learn how to think, not just what to say.
Result: The robot judges started agreeing much more with human judges.

Method B: The "Better Manual" Approach (Refining Codebooks)

For expensive, closed robots (like the ones you can't reprogram, e.g., GPT-4 or Claude), you can't change their brain. So, the researchers changed the instruction manual (the codebook) they give them.

Before: The manual said: "Rate complexity. 1 is simple, 5 is complex." (Too vague!).
After: The researchers looked at all the "Ghost Thoughts" and wrote a new manual that said: "To give a 3, look for a story that has a few ideas but they don't connect well. If the story jumps around without a clear path, that's a 3."
The Analogy: Instead of telling a new employee "Do a good job," you give them a checklist with specific examples of what "good" looks like, based on what the best employees actually did.
Result: Even different robots, using this new, clearer manual, started agreeing with each other and with humans much better.

4. Why This Matters

The paper shows that we don't need to pay humans to write down their thoughts for every single piece of content. We can use AI to infer those thoughts from the scores they already gave.

The Big Win: We can turn a boring pile of "Score: 3" data into a rich, educational resource full of "Here is why it's a 3" reasoning.
The Impact: This makes AI judges much more reliable, especially for subjective things like art, stories, or feelings, where there is no single "right" answer.

Summary in One Sentence

The researchers taught AI judges to think like humans by using a "Ghost Writer" AI to guess the reasoning behind human scores, then used those guesses to either train new AI judges or rewrite the instruction manuals for existing ones, making them much smarter and more consistent.

1. Problem Statement

Large Language Models (LLMs) are increasingly used as automated raters for subjective evaluation tasks (e.g., story quality, essay scoring, translation). However, their reliability often lags behind human judgment, particularly in tasks requiring nuanced reasoning.

The Gap: Existing annotation datasets typically contain only labels (e.g., a score of 3/5) without the thinking traces (the step-by-step reasoning process) that human experts use to arrive at those labels.
The Consequence: Without these traces, LLM raters lack the "human priors" necessary to understand why a specific rating was given, leading to poor alignment with human evaluators and inconsistent results across different models.
The Challenge: Collecting high-quality thinking traces from humans is prohibitively expensive and time-consuming, as it requires annotators to articulate complex cognitive processes rather than just selecting a label.

2. Methodology

The authors propose a human-LLM collaborative framework to infer latent thinking traces from existing label-only datasets. The core innovation is using Reasoning Language Models (RLMs) as proxies to reconstruct human reasoning.

A. Inferring Latent Thinking Traces (Rejection Sampling)

The framework uses a Rejection Sampling method to generate high-quality reasoning traces:

Input: A dataset $D_{human}$ containing inputs ( $x_i$ ) and human-annotated labels ( $y_i$ ).
Generation: An RLM (e.g., DeepSeek-R1) is prompted with the input and the annotation codebook to generate $k$ candidate thinking traces ( $t'_{i,j}$ ), each concluding with a predicted label ( $y'_{i,j}$ ).
Filtering (Rejection): Only traces where the model's predicted label matches the ground-truth human label ( $y'_{i,j} = y_i$ ) are retained.
Selection: The first matching trace is selected as the reconstructed thinking trace ( $t'_i$ ).
Output: An augmented dataset $D_{reason} = \{(x_i, t'_i, y_i)\}$ , where every data point includes the input, the inferred reasoning, and the final label.

B. Two Complementary Applications

The inferred traces are applied in two distinct ways to improve LLM raters:

Fine-Tuning Open-Weight LLMs (Reasoning-Enhanced SFT):
- The augmented dataset $D_{reason}$ is used to Supervised Fine-Tune (SFT) open-weight models.
- Unlike standard SFT (which only learns input $\to$ label), this approach trains the model to generate the full reasoning trace before the label.
- Objective: Teach the model how to reason, not just what the answer is.
Refining Annotation Codebooks for Proprietary Models:
- For black-box models (e.g., GPT-4, Claude) where weights cannot be updated, the inferred traces are used to synthesize better annotation guidelines.
- Process:
  - Instruction Refinement: The model summarizes common reasoning patterns from the traces into explicit, step-by-step instructions.
  - Rubric Enrichment: The model extracts representative critiques from the traces to create concrete, example-based scoring rubrics.
- This creates a "refined codebook" ( $C'$ ) that serves as a superior prompt for proprietary LLMs.

3. Key Contributions

Novel Framework: A scalable method to reconstruct human thinking traces from label-only data using RLMs and rejection sampling.
Dual-Application Strategy: Demonstrating that inferred traces can both fine-tune open models and refine prompts/codebooks for proprietary models.
Codebook Automation: A novel pipeline to automatically transform vague annotation guidelines into detailed, step-by-step procedures grounded in successful reasoning examples.
Post-Hoc Adaptation: Showing that the framework can even work with models that do not expose internal thinking tokens by using "post-hoc" explanations (generating reasoning after the fact based on the known label).

4. Experimental Results

The framework was evaluated across five diverse tasks: Short Story Evaluation, Essay Scoring, News Summarization, Machine Translation, and Chatbot Response Evaluation.

A. Agreement with Human Judgments

Fine-Tuning: Reasoning-enhanced SFT significantly improved the alignment between LLM raters and human judgments.
- Metric: Kendall's $\tau$ (ranking correlation) increased from a baseline of 0.197 to 0.281 (a 42.6% relative improvement) on average.
- MSE: Mean Squared Error decreased significantly across most tasks.
Codebook Refinement: Using refined codebooks with proprietary models also improved human-LLM agreement.
- Metric: Kendall's $\tau$ improved from 0.316 to 0.361 (14.2% improvement) using DeepSeek traces.
- Comparison: Prompting advanced proprietary models with refined codebooks often outperformed fine-tuned smaller models.

B. Inter-Rater Reliability (Consistency)

The refined codebooks significantly increased the agreement (ICC3) between different LLM raters.
Result: Average inter-rater reliability increased from 0.580 to 0.621.
Insight: On highly subjective tasks, the refined codebook sometimes reduced superficial agreement caused by shared biases, leading to a more natural distribution of judgments that better reflects human variance.

C. Ablation Studies

Necessity of Traces: Models trained only on labels (without traces) performed worse than the baseline, confirming that the reasoning process is critical for learning evaluation tasks.
Component Analysis: Both instruction refinement and rubric enrichment contributed to performance, but their impact varied by task (e.g., rubric refinement helped more on summary consistency, while instruction refinement helped more on chatbot correctness).

5. Significance and Impact

Unlocking Legacy Data: The paper demonstrates that vast existing "label-only" datasets can be transformed into "reasoning-rich" resources without new human annotation costs.
Bridging the Gap: It provides a practical solution to the "black box" problem of LLM evaluation, making automated raters more interpretable and reliable.
Scalability: The rejection sampling method allows for the mass generation of high-quality training data and guidelines, democratizing the ability to build specialized, high-performance evaluators.
Future Directions: The work suggests a new paradigm for human-AI collaboration where humans provide the "gold standard" label, and AI handles the cognitive heavy lifting of articulating the reasoning, potentially accelerating the onboarding of human annotators as well.

In conclusion, the paper establishes that inferred thinking traces serve as a high-fidelity proxy for human reasoning, effectively solving the reliability bottleneck in automated LLM evaluation for subjective tasks.