The Holistic Storage of Verb+Up Phrases in Text-based and Audio-based Language Models

Technical Summary: The Holistic Storage of Verb+Up Phrases in Text-based and Audio-based Language Models

Problem Statement
A central debate in linguistics concerns the trade-off between computation (applying abstract rules to generate novel representations) and storage (retrieving holistic representations from memory). While human language processing clearly utilizes both, the factors driving when a form is stored versus computed remain poorly understood. Usage-based theories suggest that high-frequency and high-predictability multi-word units are stored holistically, a phenomenon evidenced in human psycholinguistics (e.g., slower detection of particles in high-frequency phrasal verbs). However, it is unclear whether Large Language Models (LLMs) and Automatic Speech Recognition (ASR) models develop analogous holistic representations. Specifically, do these models rely on memorization for frequent phrases, and does this storage mechanism emerge across different modalities (text vs. audio) and model scales?

Methodology
The study probes internal representations in three text-based models and one audio-based model to test for holistic storage of Verb+up (V+up) phrases.

1. Models Examined:
   • Text-based LLMs: Three autoregressive models trained on the BabyLM v3 corpus (150M tokens, comparable to human exposure) with 125M, 350M, and 1.3B parameters, plus a larger 7B parameter model (OLMo-3 7B) trained on Dolma v1.7.
   • Audio-based Model: Whisper-small, an encoder-decoder ASR model trained on spoken audio.
2. Experimental Design:
   • Classifier Approach: Logistic regression classifiers were trained to distinguish the embedding of the token up when it appears as a standalone preposition from other tokens in the same sentence.
   • Testing: These classifiers were applied to V+up phrases (e.g., pick up) varying in frequency (raw corpus count) and predictability (log-odds of up following a specific verb).
   • Metric: The classifier's logit score indicates similarity to the standalone up representation. A lower logit score implies the representation of up within the phrase has diverged from the standalone form, suggesting holistic storage.
   • Experiments:
     • Experiment 1: Tested up as a preposition only.
     • Experiment 2: Tested up as a subword unit (including instances embedded in words like update) to rule out semantic bleaching as the sole cause of divergence.
     • Experiment 3: Applied the same methodology to Whisper-small using spoken audio segments, extracting representations from both the encoder and decoder.
3. Analysis: Bayesian mixed-effects regression and Generalized Additive Models (GAMs) were used to analyze logit scores across frequency, predictability, and hidden layers.

Key Contributions

• Open-Source Models: The authors trained and released three autoregressive models on the BabyLM v3 corpus, checkpointed every 20M tokens, to facilitate research on human-scale language learning.
• Cross-Modal Evidence: The study establishes that holistic phrasal storage emerges in both text-based LLMs and ASR models, demonstrating that frequency- and predictability-driven representations arise across modalities.
• Scale and Capacity: The research disentangles the effects of model size, showing that while frequency effects are robust across sizes, sensitivity to predictability strengthens with scale, suggesting that capturing co-occurrence statistics beyond raw frequency requires greater representational capacity.

Results

• Holistic Storage: All models showed evidence of holistic storage. High-frequency and high-predictability V+up phrases resulted in significantly lower classifier logit scores, indicating that the representation of up in these contexts diverges from the standalone token.
• Frequency vs. Predictability:
  • Frequency: Strong, consistent negative effects were observed across all models and sizes.
  • Predictability: Effects were weaker in smaller models (BabyLMs) but became progressively stronger and more negative in larger models (OLMo-3 7B). This suggests that sensitivity to predictability requires larger model capacity.
• Layer Dynamics: In larger models, the divergence in representations (holistic storage) appeared in earlier hidden layers. In smaller models, this effect emerged only in later layers.
• Audio Modality: Whisper-small exhibited similar patterns in both its encoder and decoder. High-frequency and high-predictability phrases showed lower logit scores, mirroring the text-based results. The encoder's behavior suggests it abstracts acoustic variability to form representations similar to those in text-based models.
• Subword Robustness: Experiment 2 confirmed that these effects persist even when up is treated as a subword unit, ruling out semantic bleaching as the sole explanation for the divergence.

Significance and Claims
The paper argues that these findings support usage-based theories of language, which posit that holistic storage is a gradient phenomenon driven by frequency and predictability rather than a binary distinction between stored items and computed rules.

• Human-Scale Learning: The emergence of holistic storage in models trained on data amounts comparable to human exposure (BabyLMs) suggests that such storage is a natural consequence of learning from distributional patterns, not a result of massive-scale memorization.
• Continuum of Representation: The by-layer results suggest that holistic storage is better understood as a spectrum. Representations may remain compositional in lower layers and become increasingly holistic in higher layers, with larger models exhibiting this divergence earlier.
• Transformer Utility: The study positions transformer architectures as productive frameworks for investigating the mechanics of human-like storage patterns, offering a level of mechanistic detail (internal representations) that behavioral data alone cannot provide.

The authors maintain modesty regarding limitations, noting the study is restricted to English and the V+up construction, and that future work is needed to examine storage emergence as a function of training dynamics rather than just final checkpoints.