💻 computer science

Black Big Boxes: Tracing Adjective Order Preferences in Large Language Models

This paper investigates how large language models acquire English adjective ordering preferences, finding that while their predictions largely reflect training data frequencies, they also demonstrate robust generalization to unseen combinations and effectively leverage contextual cues beyond simple memorization.

Original authors: Jaap Jumelet, Lisa Bylinina, Willem Zuidema, Jakub Szymanik

Published 2026-02-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jaap Jumelet, Lisa Bylinina, Willem Zuidema, Jakub Szymanik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to speak English. You want to know if the robot is actually "learning" the rules of how to put words together, or if it's just acting like a super-powered parrot that memorized every sentence it ever heard.

This paper investigates that question by looking at adjective order—the way we naturally stack descriptive words before a noun. For example, we say "a big red box," not "a red big box." If you swap them, it sounds slightly weird, even though it's not strictly "wrong."

Here is what the researchers found, explained through simple analogies:

1. The Robot is a "Statistical Parrot" (Mostly)

The researchers trained a large language model (a type of AI) and then tested it on thousands of adjective pairs. They asked: Does the robot prefer "big red" over "red big"?

The Finding: The robot is incredibly good at this. In fact, its preferences are almost entirely explained by how often it saw those words together in its training data.

The Analogy: Imagine a chef who has cooked 10,000 meals. If they always put salt before pepper in their recipes, they will instinctively do the same in a new dish. The robot isn't necessarily understanding why salt goes before pepper; it just knows that "salt-pepper" is a very common pattern in the books it read.
The Evidence: The researchers found a direct line between the frequency of word pairs in the training data and the robot's choices. If the data says "corporate social" is common, the robot picks it. If "social corporate" is rare, the robot avoids it.

2. The Robot Can "Guess" New Combinations

Here is the twist. If the robot were just a parrot memorizing a list, it should fail when it sees a combination of words it has never seen before.

The Finding: The robot actually does a pretty good job with brand-new adjective pairs it hasn't memorized.

The Analogy: Imagine the chef has never made a "spicy sweet" dish before. But because they know how "spicy" and "sweet" usually behave in other dishes, they can guess the right order for a new recipe.
The Evidence: Even when the robot encountered adjective pairs that appeared zero times in its training data, it still chose the natural order about 85% of the time. This suggests it has learned some underlying "feel" for word order, not just a list of memorized sentences.

3. Context is the "Magic Spotlight"

The researchers also tested the robot with and without the rest of the sentence (the context).

The Finding: Giving the robot the full sentence makes it much smarter.

The Analogy: Imagine you hear someone say, "Take the wooden large box." Without context, this sounds wrong. But if the sentence is, "Take the wooden large box, not the plastic one," it suddenly makes perfect sense. The word "plastic" creates a contrast that forces the order to flip.
The Evidence: The robot uses clues from earlier in the sentence to decide the order. The researchers used a "flashlight" technique (called feature attribution) to see which words the robot was looking at. They found two types of clues:
1. Local Clues: Words right next to the adjectives (like "the" or "a") that hint at the order.
2. Semantic Clues: Words far back in the sentence that set a theme (like contrasting "wooden" with "plastic").

4. How the Robot "Grows" Its Knowledge

The researchers watched the robot learn step-by-step during its training. They found three distinct stages:

The Blank Slate: At the very beginning, the robot has no idea about word order.
The Speed Run: Very quickly (in just a tiny fraction of its total training time), it learns the basic patterns of how words usually sit next to each other.
The Refinement: After the basics are down, it starts learning how the surrounding sentence changes the rules.

The Bottom Line

This paper doesn't claim the robot is a human linguist. Instead, it shows that the robot is a hybrid learner:

It relies heavily on statistics (memorizing what it has seen).
But it also has a generalizing ability (making smart guesses about things it hasn't seen).
And it uses context (the surrounding story) to fine-tune its choices.

The authors conclude that while we can't say the robot "understands" language like a human does, it has successfully built a complex internal map of how English words fit together, driven by a mix of memorization and pattern recognition.

Technical Summary: Black Big Boxes: Tracing Adjective Order Preferences in Large Language Models

Problem Statement

Adjective order in noun phrases (e.g., large wooden box vs. wooden large box) represents a complex linguistic phenomenon characterized by graded, context-sensitive preferences rather than binary grammatical constraints. While extensively studied in theoretical linguistics, the mechanisms driving these preferences remain difficult to capture due to their reliance on multiple interacting factors (semantic, pragmatic, information-theoretic) and the fact that even the best multi-factor models explain only ~71% of human preferences.

This paper investigates how Large Language Models (LLMs) acquire and utilize these "soft" word order preferences. The central research question is the extent to which LLM behavior regarding adjective order can be explained by distributional learning (memorization of surface co-occurrence patterns) versus the emergence of generalizable, context-sensitive linguistic behavior. Specifically, the authors ask:

To what extent are LLM adjective order preferences (AOPs) driven by training data frequencies?
Do models generalize to unseen adjective combinations beyond direct memorization?
How does sentence context influence these preferences, and what specific cues drive them?

Methodology

The authors employ a systematic empirical analysis using the Pythia suite of language models (ranging from 70M to 12B parameters), all trained on The Pile corpus. The study utilizes intermediate training checkpoints to analyze learning dynamics at a fine-grained level.

Data and Metrics

Evaluation Corpus (CAP): The study uses the Corpus of Adjective Pairs (CAP), derived from the BabyLM 100M corpus. It contains 9,396 sentences with double adjectives modifying a noun, alongside their "swapped" counterparts.
Metrics:
- AOP-∆: The difference in log probability between the natural order ( $A_1A_2N$ ) and the swapped order ( $A_2A_1N$ ), calculated both in isolation and with context.
- AOP-%: The percentage of items where the model prefers the natural order (AOP-∆ > 0).
Distributional Analysis: The authors utilize Infini-gram to retrieve exact n-gram statistics (uni-, bi-, and trigrams) from The Pile, allowing for a direct comparison between model predictions and training data frequencies.
Generalization Test: To assess memorization vs. generalization, the authors analyze model performance on adjective pairs that have been seen $0$ times (unseen) versus those seen $1$ to $10$+ times during training, using intermediate checkpoints.
Contextual Analysis:
- Random Contexts: Comparing AOP-∆ with true sentence context against an expectation over random contexts to quantify "contextual gain."
- Feature Attribution: Using Integrated Gradients to identify specific tokens in the context that contribute most to the model's preference for a specific adjective order. These are mapped to Part-of-Speech (POS) tags to identify global patterns.

Key Contributions and Results

1. Acquisition Dynamics and Model Size

Performance: LMs exhibit strong and consistent preferences for naturally occurring adjective orders. Contextual AOP-% increases with model size, rising from 85.0% (70M) to 91.7% (12B).
Learning Phases: Analysis of intermediate checkpoints reveals three distinct phases of acquisition:
1. Initialization: AOP judgments are not yet formed.
2. Acquisition: Rapid formation of AOP judgments; sensitivity to adjective order emerges early (around 1.5% of total training).
3. Consolidation: Judgments stabilize, and the role of context is reinforced.
Token-Level Localization: The largest probability difference between natural and swapped orders occurs at the second adjective token. However, context influences the probability of the first adjective immediately, suggesting context shapes preferences from the onset of the phrase.

2. The Role of Training Data Statistics

Correlation with Frequencies: There is a strong correlation between model AOP-∆ and bigram frequencies in the training data. Simple bigram statistics account for a large portion of model behavior (86.8% predictive accuracy for bigram counts).
Limits of Memorization: While bigram statistics are highly predictive, they do not fully explain model performance. Trigram statistics (adjective-adjective-noun) show lower predictive accuracy (76.7%), and models still perform well on items where the specific trigram is rare or unseen in the training data.

3. Generalization Beyond Memorization

Zero-Shot Performance: Despite the strong influence of distributional statistics, models exhibit robust generalization. On adjective pairs never seen during training (0 occurrences), models still achieve high AOP-% scores (reaching ~85% in some splits).
Interpretation: This indicates that model behavior cannot be reduced solely to the memorization of observed orders; models generalize to unseen combinations, likely leveraging learned patterns from other adjective orders.

4. The Role of Context

Contextual Gain: Adding sentence context significantly improves AOP predictions. In 67.6% of cases, context increases the AOP-∆ score.
Semantic vs. Random: True context improves predictions in 65.1% of cases, whereas random contexts only improve predictions in 39.6%, confirming that the semantic content of the context is the driver, not just the presence of linguistic signal.
Sources of Influence: Feature attribution identifies two primary sources of contextual influence:
1. Local Collocational Cues: Determiners, adverbs, and prepositions immediately preceding the adjective phrase act as strong drivers (e.g., a determiner co-occurring frequently with a specific adjective type).
2. Earlier Semantic Cues: Adjectives and nouns appearing earlier in the sentence bias ordering based on semantic similarity. The top-scoring context adjectives show higher cosine similarity to the first adjective in the phrase than the second.

Significance and Claims

The paper positions LMs not as explanatory theories of human adjective order, but as empirical tools for probing the sources of information that influence word order preferences.

Beyond Surface Statistics: The study demonstrates that while LMs heavily rely on training data frequencies (distributional learning), they also exhibit generalization capabilities that go beyond simple memorization of co-occurrences.
Context Sensitivity: The research highlights that LMs leverage rich sentential context to resolve graded preferences, a feature often absent in traditional theoretical accounts of adjective order which focus on adjectives in isolation.
Methodological Contribution: By combining behavioral analysis with feature attribution, the authors identify specific contextual cues (local collocational and distant semantic) that modulate word order. These findings suggest new avenues for empirical study in human language processing, particularly regarding the role of context in "soft" linguistic constraints.
Limitations: The authors acknowledge that their findings are specific to English and do not constitute a complete explanatory account of human linguistic behavior. They note that the trade-off between memorization and generalization in humans remains an open empirical question requiring further data.

In summary, the paper provides a behavioral characterization of how LMs acquire adjective order preferences, tracing them to a combination of training data statistics, generalization mechanisms, and specific contextual cues, thereby offering a framework for studying word order in language models.