Is my model perplexed for the right reason? Contrasting LLMs' Benchmark Behavior with Token-Level Perplexity

1. Problem Statement

Current Large Language Model (LLM) evaluations rely heavily on task performance benchmarks (e.g., accuracy on classification or generation tasks). The authors argue that these metrics suffer from two critical flaws:

• Lack of Mechanistic Insight: High accuracy does not guarantee that the model is using the correct linguistic reasoning. A model might achieve high scores via "shortcuts" or heuristics unrelated to the intended linguistic feature (the "Chinese Room" problem).
• Confirmation Bias: Researchers often interpret correct outputs as evidence of genuine understanding, projecting their own expectations onto the model's behavior without verifying the underlying decision-making process.

Existing Explainable AI (XAI) methods, such as feature attribution (e.g., saliency maps), are often unstable and can be misleading. The paper seeks a more principled, stable method to determine if a model is "perplexed for the right reason"—i.e., whether its internal probability shifts align with specific, linguistically pivotal tokens.

2. Methodology

The authors propose a token-level perplexity (PPL) framework to test specific hypotheses about model behavior without relying on unstable attribution techniques.

Experimental Setup: Minimal Pairs

The study utilizes minimal pairs of prompts. Each pair consists of two sentences that are identical except for one or a few "pivotal tokens" that determine the grammaticality, ambiguity, or semantic validity of the sentence.

• Example: "Andrei approached the person with a green chair" (ambiguous) vs. "Andrei approached the person who had a green chair" (unambiguous).
• The model is asked to identify which sentence is correct/ambiguous.

Metrics

The authors define two primary metrics to analyze the model's internal state, independent of its final task accuracy:

1. Accuracy: Standard binary classification (does the model assign lower sequence-level PPL to the "correct" prompt?).
2. Token-Level Perplexity Metrics: To measure why the model made its choice, they calculate the difference in perplexity ($\Delta PPL$) between the correct and incorrect prompts.
   • Plain Proportion: The ratio of the absolute PPL difference contributed by pivotal tokens to the total sequence $\Delta PPL$.
     • Formula: $\frac{\sum |PPL_{pivotal}^{correct} - PPL_{pivotal}^{incorrect}|}{\sum |PPL_{all}^{correct} - PPL_{all}^{incorrect}|}$
     • Goal: A value of 1.0 implies the pivotal tokens explain the entire decision.
   • Normalized Proportion: A refined metric that accounts for the direction of the effect and normalizes against the number of tokens. It subtracts the median proportion of non-pivotal tokens from the average proportion of pivotal tokens.
     • Goal: A value of 1.0 indicates pivotal tokens drive the shift in the correct direction; 0 indicates they are no more informative than average tokens.

Models and Tasks

• Models: Four open-weight base models: Gemma3-4B, Mistral0.3-7B, Llama3.2-3B, and Qwen2.5-7B.
• Tasks:
  1. Sanity Check: Nonsense words (one real word inserted).
  2. Stereotypes: CrowS-pairs dataset (social bias detection).
  3. Linguistic Phenomena: BLiMP dataset (Anaphor agreement, Animacy verification) and DUST dataset (Ambiguity detection).

3. Key Results

The experiments reveal a significant disconnect between task accuracy and the underlying reasoning mechanisms.

• Sanity Check (Nonsense Words): The method works as intended. In this controlled setting, the pivotal token (the single real word) accounts for the vast majority of the $\Delta PPL$, validating the framework's ability to detect "correct" reasoning.
• Main Tasks (Linguistic & Stereotypes):
  • Decoupling of Accuracy and Reasoning: Models often achieve high accuracy on tasks (e.g., Anaphor agreement, Ambiguity) but fail to attribute the perplexity shift to the correct pivotal tokens.
  • Incomplete Explanation: Even when models perform well, the pivotal tokens never fully explain the perplexity shift.
    • In DUST (Ambiguity), pivotal tokens drive the shift most strongly, sometimes explaining nearly 100% of the difference.
    • In BLiMP (Anaphor/Animacy) and Stereotypes, pivotal tokens explain at most ~50% of the $\Delta PPL$.
  • Alternative Cues: Other tokens, such as punctuation (periods) or surrounding context, contribute significantly to the decision-making process. For example, in the Stereotypes task, periods contributed to the PPL difference across all models.
  • Model Variability: Different models exhibit different heuristics. For instance, Mistral attributes less explanatory power to periods in the Stereotypes task compared to Gemma, Llama, or Qwen.

4. Key Contributions

1. Principled Interpretability Framework: Introduction of a simple, stable method using token-level perplexity to test specific hypotheses about model behavior, avoiding the instability of feature-attribution methods.
2. New Metrics: Development of "Plain Proportion" and "Normalized Proportion" metrics to quantify the extent to which specific tokens drive model decisions.
3. Empirical Evidence of Heuristic Reliance: Demonstration that current LLMs, even when achieving high benchmark scores, often rely on heuristics other than the linguistically expected cues. They are often "perplexed for the wrong reasons."
4. Resource Release: Public release of code and data for the "Semantic match DUST" framework.

5. Significance and Implications

• Critique of Benchmarks: The paper challenges the validity of standard benchmarks as proxies for linguistic competence. High accuracy does not equate to understanding; models may be "right for the wrong reasons."
• Need for Better Evaluation: It underscores the necessity of interpretability methods that can verify how a model reaches a conclusion, not just what the conclusion is.
• Diagnostic Tool: The proposed metrics serve as a diagnostic tool to identify when models are relying on spurious correlations (e.g., punctuation or specific context words) rather than the intended linguistic features.
• Future Directions: The authors suggest that future model development and evaluation must move beyond downstream task performance to ensure models align with human linguistic expectations and reasoning mechanisms.

Limitations Noted by Authors

• Generalizability: Results are specific to the datasets and models tested; tasks where "pivotal tokens" are hard to define (e.g., complex semantic reasoning) may not fit this framework.
• PPL Definition: The results depend on the sequential definition of perplexity; alternative calculation methods might yield different insights.
• Prompt Sensitivity: While some prompt variations were tested, the sensitivity of results to prompt phrasing requires further investigation.