The Frequency Confound in Language-Model Surprisal and Metaphor Novelty

Technical Summary: The Frequency Confound in Language-Model Surprisal and Metaphor Novelty

Problem Statement
Language-model (LM) surprisal is frequently utilized as a proxy for contextual predictability and has been reported to correlate with human judgments of metaphor novelty. Recent studies suggest an "inverse scaling effect," where smaller LMs or models at earlier training stages exhibit stronger correlations between surprisal and human behavioral data (such as reading times or novelty ratings) than larger, fully trained models. However, a critical confounding factor remains unclear: the tight interdependence between surprisal and lexical frequency. It is uncertain whether the observed correlations reflect genuine contextual predictability or are merely artifacts of lexical frequency, which is known to strongly influence both processing difficulty and novelty judgments. This paper investigates whether the "optimal" surprisal settings (small models, early training) genuinely capture human-like contextual prediction or if they simply align more closely with lexical frequency.

Methodology
The authors conducted a comprehensive analysis using the VU Amsterdam Metaphor Corpus (VUAMC), which contains 15,155 metaphoric content words with associated human novelty ratings (scaled from -1 for conventional to +1 for novel).

• Model Suite: The study utilized the Pythia model family (Biderman et al., 2023), comprising eight model sizes ranging from 70M to 12B parameters. All models were trained on the same 300B-token corpus (The Pile).
• Training Dynamics: To analyze pretraining progress, the authors examined 154 intermediate checkpoints for the Pythia-70M model, capturing the evolution of the model from early training (1 step) to full pretraining (300B tokens).
• Metrics:
  • Surprisal: Calculated as $-\log p(w_i | w_{<i})$ for target metaphoric words within their sentence context, with corrections for tokenization and leading whitespace.
  • Lexical Frequency: Two distinct measures were employed to estimate word frequency:
    1. NLF-Human: Negative log frequency derived from wordfreq, representing an aggregate of general language use (approximating an "average English speaker").
    2. NLF-LM: Negative log frequency calculated specifically from the subtoken sequences observed in the Pythia pretraining data up to each specific checkpoint.
• Analysis: The study quantified associations using Spearman's ($\rho$) and Pearson's ($r$) correlation coefficients, as well as the Area Under the ROC Curve (AUC) for classifying novel metaphors. The analysis focused on the relationships between: (1) Surprisal and Novelty, (2) Frequency and Novelty, and (3) Surprisal and Frequency.

Key Results

1. Frequency as a Stronger Predictor: Across all model sizes and training stages, lexical frequency (both NLF-Human and NLF-LM) consistently demonstrated a stronger association with metaphor novelty ratings than LM surprisal. NLF-Human yielded the highest correlations ($\rho \approx 0.66$, AUC $\approx 0.90$).
2. Inverse Scaling and Training Dynamics:
   • Model Scale: The correlation between surprisal and novelty decreased as model size increased. Smaller models (70M) showed the strongest surprisal–novelty association.
   • Training Progress: For Pythia-70M, the surprisal–novelty association was weak initially, rose sharply after 64 training steps, and peaked at 128 steps (approx. 268M tokens seen), reaching $\rho \approx 0.62$. Beyond this peak, the association declined.
3. The Frequency Confound: Crucially, the trends in the surprisal–novelty association mirrored the trends in the surprisal–frequency association.
   • At the early training stage (128 steps) where surprisal correlated most strongly with novelty, it also correlated most strongly with frequency ($\rho \approx 0.95$ with NLF-LM).
   • As models grew larger or training progressed further, the correlation between surprisal and frequency weakened, paralleling the decline in the surprisal–novelty correlation.
4. Alignment: The configurations where surprisal performed "best" (small models, early training) were precisely those where surprisal was most tightly aligned with lexical frequency.

Significance and Claims
The paper argues that the often-reported optimal settings for LM surprisal (small models or early training stages) may not provide straightforward evidence for surprisal-based accounts of metaphor novelty or processing difficulty. Instead, the strong correlations observed in these settings likely reflect the model's reliance on lexical frequency rather than genuine contextual predictability.

The authors conclude that:

• Lexical frequency is a major underlying factor in metaphor novelty judgments, often acting as a stronger predictor than contextual surprisal.
• The "inverse scaling effect" observed in surprisal–novelty correlations may be a byproduct of the changing relationship between surprisal and frequency across model scales and training stages.
• Researchers should exercise caution when interpreting strong early-training or small-model surprisal–novelty associations as direct support for theories of processing difficulty based on contextual predictability.
• Future progress in this field requires a clearer theoretical and methodological separation between contextual predictability and lexical-frequency effects.

The paper does not claim that larger models produce "poor" estimates of predictability, but rather that the specific conditions under which surprisal aligns with human novelty ratings are confounded by frequency. The authors call for further development of human ratings that better reflect theories of novelty and originality, distinct from simple frequency effects.