What Is Missing: Interpretable Ratings for Large Language Model Outputs

This paper introduces the "What Is Missing" (WIM) rating system, which converts natural language feedback describing output deficiencies into interpretable scalar ratings via sentence embedding similarity, thereby improving preference learning signals and enabling qualitative debugging compared to traditional discrete numerical ratings.

The Gist

Here is an explanation of the paper "What Is Missing (WIM)" using simple language and creative analogies.

The Big Problem: The "Vague Score" Trap

Imagine you are a chef trying to improve your cooking. You ask a food critic to taste your soup and tell you how it is.

• The Old Way (Numerical Ratings): The critic says, "This soup is a 7 out of 10."
  • The Problem: You don't know why it's a 7. Is it too salty? Did you forget the garlic? Is the texture weird? If you get another soup rated a 7, you have no idea how it compares to the first one. They are both just "okay." This makes it hard for you to learn exactly what to fix.
• The New Way (WIM): The critic says, "This soup is a 7, but here is exactly what is missing: It needs more salt and a pinch of black pepper."
  • The Benefit: Now you know exactly what to change. You can fix the salt, add the pepper, and try again.

The Paper's Goal: The authors (Nicholas Stranges and Yimin Yang) realized that Large Language Models (LLMs) are like those chefs. When we train them, we usually give them vague "7 out of 10" scores. They want to switch to a system where the "critic" tells the model exactly what is missing from its answer.

---

How the "What Is Missing" (WIM) System Works

The paper introduces a clever trick to turn those vague scores into precise, helpful feedback. Here is the process, broken down into a story:

1. The Performance (The Model's Output)

The AI writes an answer to a question. Let's say it writes a story about a space adventure.

2. The Critic's Job (The "Missing" List)

Instead of just giving a number, a human or another AI (the "Judge") looks at the story and writes a short note describing what is missing.

• Example: "The story forgot to mention how the spaceship's engine works, and it didn't explain why the alien was scared."

3. The Magic Translation (The Embedding)

This is the technical part made simple. The computer takes two things:

1. The Story the AI wrote.
2. The "Missing" Note the Judge wrote.

It turns both of these into invisible "fingerprint" vectors (mathematical representations of meaning).

4. The Similarity Score (The Rating)

The computer compares the "Story Fingerprint" and the "Missing Note Fingerprint."

• If the Note says "Nothing is missing": The two fingerprints look almost identical. The computer gives a perfect score (1.0).
• If the Note says "You forgot the engine and the fear": The fingerprints are very different. The computer gives a lower score.

The Analogy: Imagine the AI's answer is a puzzle. The "Missing" note is a list of the pieces you forgot to put in.

• If you forgot one piece, the list is short, and your score is high.
• If you forgot half the puzzle, the list is long, and your score is low.
• The computer measures the "distance" between the puzzle you built and the list of missing pieces.

---

Why Is This Better? (The "Learning Signal")

The paper argues that this system fixes two big problems with the old way of training AI:

1. No More "Ties" (The Flatline Problem)

• Old Way: In a race, if two runners both get a "7/10" time, the computer doesn't know who is faster. It's a tie. The AI gets confused and doesn't learn which answer was actually better.
• WIM Way: Because the "Missing" notes are unique sentences, the scores are rarely the same. One answer might be "8.4" and the other "7.1". The AI sees a clear difference and knows exactly which one to copy. It's like having a ruler with tiny millimeter marks instead of just big inch marks.

2. You Can See the Mistakes (Interpretability)

• Old Way: If an AI fails, you just see a low number. You have to guess why.
• WIM Way: If the AI gets a low score, you can look at the "Missing" note and say, "Ah! The judge said it forgot to mention the legal basis for the law." Now you can debug the system easily. It's like getting a teacher's red pen comments instead of just a grade of "F".

---

The Results: Did It Work?

The authors tested this on a model called Llama-3.

• They trained one version using the old "7 out of 10" scores.
• They trained another version using the new "What Is Missing" scores.

The Winner: The "What Is Missing" version learned faster and became better at its job. It made fewer mistakes and was more confident in its answers.

The "Self-Judge" Twist

The paper also tried something cool: Self-Reflection.
Instead of a human or a different AI acting as the judge, they let the AI being trained act as its own critic.

• Analogy: It's like a writer writing a draft, then reading it over and saying, "I forgot to explain the ending," before writing the next draft.
• They found that having a fixed judge (a stable, unchanging AI) worked better than a moving judge (the AI judging itself while it was still learning), because a changing judge can be confusing.

Summary

The paper proposes that instead of telling an AI "You did okay (7/10)," we should tell it "You did okay, but you forgot X, Y, and Z." By turning those "forgotten things" into a mathematical score, the AI learns much faster, makes fewer mistakes, and we humans can actually understand why it made those mistakes.

It turns the black box of AI training into a transparent, helpful conversation.

Technical Summary

Here is a detailed technical summary of the paper "What Is Missing: Interpretable Ratings for Large Language Model Outputs."

1. Problem Statement

Current preference learning methods for Large Language Models (LLMs), such as Proximal Policy Optimization (PPO) and Direct Preference Optimization (DPO), rely on ranking model outputs based on human or AI judge feedback. The paper identifies two critical shortcomings in existing rating systems (direct rankings and discrete numerical scales like 1–10):

• Low Interpretability: A single numerical score or a rank order does not explain why an output was preferred. This lack of transparency makes it difficult to debug preference labels or understand specific failure modes.
• Weak Learning Signals: Numerical rating systems are discrete and often result in "ties" (where two different outputs receive the exact same score). In pairwise preference learning, if the winning and losing outputs have identical ratings, no gradient signal is generated to update the model. The paper notes that in standard numerical systems, nearly 43% of pairwise comparisons result in ties, severely limiting the efficiency of the training process.

2. Methodology: The WIM System

The authors propose What Is Missing (WIM), a rating system that converts natural language feedback into a scalar score using semantic embeddings.

Core Mechanism

1. Feedback Generation: Instead of assigning a number, a judge (human or LLM) analyzes a model output ($s_1$) and writes a natural language description ($s_2$) of what is missing from the response (e.g., missing facts, lack of detail, or logical gaps).
2. Embedding: Both the original output ($s_1$) and the "missing" feedback ($s_2$) are passed through a sentence embedding model (e.g., all-mpnet-base-v2) to generate high-dimensional vectors ($S_1$ and $S_2$).
3. Scoring: The WIM rating is calculated as the cosine similarity between $S_1$ and $S_2$.
   • High Similarity (Score $\approx$ 1): The "missing" text is semantically similar to the output, implying the output is complete (or the judge found nothing to add).
   • Low Similarity (Score $\approx$ -1): The "missing" text is orthogonal or antiparallel to the output, indicating significant content gaps.
   • Special Case: If the judge provides no feedback (nothing is missing), the score is manually assigned a perfect 1.

Mathematical Formulation

The score is defined as:
$$\text{WIM} = \frac{S_1 \cdot S_2}{\|S_1\| \|S_2\|}$$
The authors conceptualize the "missingness" as the orthogonal component of the feedback vector relative to the output vector. As the orthogonal component grows, the cosine similarity decreases, providing a continuous measure of quality.

Integration

WIM is algorithm-agnostic. It produces a scalar score that can be fed into any existing preference learning algorithm (DPO, PPO, GRPO) without modifying the learning algorithm itself. It can also be mixed with numerical ratings using a hyperparameter $\zeta$.

3. Key Contributions

• Interpretability: Every scalar rating is directly traceable to the specific text critique ("what is missing") that generated it, enabling qualitative debugging of the training data.
• Continuous Distribution: Unlike discrete 1–10 scales, WIM ratings are derived from a continuous cosine similarity space. This drastically reduces the frequency of ties in pairwise comparisons.
• Enhanced Learning Signals: By reducing ties and increasing the variance (delta) between winning and losing ratings, WIM provides a stronger, more consistent gradient signal for model optimization.
• Self-Judging Capability: The system supports "Moving Judges" (the model critiques its own output) and "Fixed Judges" (a frozen reference model critiques), offering flexibility in training dynamics.

4. Experimental Results

The authors fine-tuned a Meta-Llama-3-8B-Instruct model on the ultrafeedback-prompt dataset using Online Direct Preference Optimization (ODPO). They compared WIM against a standard 1–10 numerical rating system.

• Rating Distribution:

  • Numerical: Heavily clustered around 7–8; 42.78% of pairwise comparisons resulted in ties (zero delta).
  • WIM: Followed a more normal distribution with fewer ties; only 2.00% of comparisons resulted in ties.
  • Delta: The average rating delta between winning and losing pairs was 47.82% higher for WIM (1.396 vs. 0.928 for numerical).

• Training Metrics:

  • Loss Reduction: The WIM method (Fixed Judge) reduced the DPO training loss by a factor of 2.95 compared to the numerical method.
  • Entropy: WIM training resulted in a significantly larger decrease in mean entropy (-106.94 vs. -45.3 for numerical), indicating the model became more confident in its outputs.
  • Reward Advantage: WIM showed larger scaling in reward advantage trajectories compared to the near-constant trajectory of the numerical system.

• Task Performance:

  • On a held-out test set, the WIM Fixed Judge model achieved a 52.0% win rate against the base model, compared to 50.1% for the numerical method (a 3.79% relative improvement).
  • Note: While the win rate improvement was measurable, the authors note that statistical significance was not fully achieved in these specific tests, suggesting further validation is needed.

5. Significance and Future Work

The paper demonstrates that the quality of the data signal (how preferences are encoded) is as critical as the optimization algorithm itself. By shifting the focus from the algorithm to the rating mechanism, WIM offers a simple, plug-and-play improvement to existing post-training pipelines.

Limitations & Future Directions:

• Judge Reliability: The system relies on the judge's ability to follow instructions. The paper notes failure cases where the judge LLM produced nonsensical feedback (e.g., outputting "wim" instead of a critique).
• Moving Judge Instability: Using the model being trained as its own judge ("Moving Judge") resulted in slightly lower performance than a Fixed Judge, likely due to non-stationary targets during training.
• Human Validation: Future work requires testing WIM with human judges to validate inter-annotator agreement and the interpretability claims.

In conclusion, WIM provides a robust, interpretable alternative to numerical ratings that enhances the learning signal for LLM alignment, potentially leading to more efficient and effective preference learning.