💬 NLP

More Aligned, Less Diverse? Analyzing the Grammar and Lexicon of Two Generations of LLMs

This study compares two generations of LLMs against human-authored news text using HPSG formalism and diversity metrics, revealing that while human writing remains stable, newer instruction-tuned models exhibit significantly reduced syntactic and lexical diversity, suggesting a potential narrowing of expressive range despite improved coherence.

Original authors: Adrián Gude, Roi Santos-Ríos, Francis Bond, Dan Flickinger, Carlos Gómez-Rodríguez, Olga Zamaraeva

Published 2026-05-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Adrián Gude, Roi Santos-Ríos, Francis Bond, Dan Flickinger, Carlos Gómez-Rodríguez, Olga Zamaraeva

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a music critic comparing two bands: The Humans (professional journalists) and The Robots (Artificial Intelligence). You want to see how their music has changed over time.

This paper is like a deep dive into the sheet music of these two groups. Instead of just listening to the melody (the words), the researchers looked at the grammar (the chords and structure) and the vocabulary (the specific instruments used) to see if the robots are sounding more like humans or if they are becoming something else entirely.

Here is the breakdown of their findings in simple terms:

1. The Setup: Two Eras, Two Bands

The researchers compared two different time periods:

The Humans: They looked at news articles from The New York Times written in 2023 and 2025.
The Robots: They compared two generations of AI models:
- Older Robots (2023): These were the "base" models. Think of them as raw, untrained musicians who just play whatever comes to mind.
- Newer Robots (2025): These are the "instruction-tuned" models. Think of them as musicians who have been given a strict rulebook and a coach telling them exactly how to behave, what to say, and how to be "helpful."

2. The Big Discovery: "More Aligned, Less Diverse"

The main finding is a bit surprising. You might expect that as AI gets smarter and more "aligned" with human instructions, it would start sounding more like a human writer.

The Reality: The newer AI models actually sound less like humans and less like their own older versions.

The Humans: Professional journalists haven't changed much. Their writing style, sentence structure, and word choices have remained stable and diverse over the last two years. They are like a jazz band that keeps improvising with new ideas.
The Older Robots: These models were actually quite diverse. They used a wide variety of sentence structures and words, sometimes even more varied than the humans in terms of vocabulary.
The Newer Robots: These models have become boringly consistent. They have lost their "flavor."
- Syntactic Diversity (Structure): They use fewer types of sentence structures.
- Lexical Diversity (Words): They use a much narrower range of words. They avoid specific names, dates, and unique details.

3. The "Safety Filter" Effect

Why did the robots change? The paper suggests it's because of Instruction Tuning.

Imagine the older robots were wild painters who threw paint everywhere. The newer robots are painters who have been told, "Don't make mistakes, don't hallucinate, and stick to the facts."

To follow these rules, the newer AI models started playing it safe.

They stopped using specific names (like "Washington" or "Senator Ben Cardin") because they didn't want to get the facts wrong.
They stopped using complex sentence structures that might be risky.
Instead, they started using long, safe, repetitive sentences that are easy to understand but lack the "spark" of human writing.

4. The "Easy to Read" Paradox

Here is a strange twist: The newer AI models write longer sentences than humans, but those sentences are easier for a computer to analyze.

The Analogy: Imagine a human writer building a complex, winding castle with hidden doors and secret passages. It's beautiful, but hard to map out.
The newer AI builds a very long, straight hallway with no doors. It's longer than the human's castle, but it's incredibly easy to walk down because it's so predictable.

The researchers found that the newer AI text is so regular and formulaic that a computer grammar checker can process it almost perfectly (99% success), whereas human writing is slightly more chaotic and harder to parse (94-95% success).

5. The Conclusion

The paper concludes that while newer AI models are better at following instructions and sounding "helpful," they have traded creativity and variety for safety and predictability.

They are no longer "average" writers; they are becoming "safe" writers. They have narrowed their expressive range so much that they are now less diverse than both human journalists and the older, less-trained AI models.

In short: The robots got better at following the rules, but in doing so, they forgot how to be interesting.

Technical Summary: More Aligned, Less Diverse? Analyzing the Grammar and Lexicon of Two Generations of LLMs

Problem Statement
While Large Language Models (LLMs) are increasingly compared to human writers across linguistic dimensions, robust methods for quantifying these differences remain underdeveloped. Specifically, the impact of instruction tuning and Reinforcement Learning from Human Feedback (RLHF) on the grammatical and lexical diversity of model outputs is not well understood. Prior research has established that base models differ from human text, but it is unclear how newer, instruction-tuned generations compare to both humans and their own predecessors. This study addresses the gap by investigating whether newer LLMs, designed for better alignment and coherence, have sacrificed syntactic and lexical variety.

Methodology
The authors employ a formal linguistic framework rather than standard NLP task-based metrics to ensure robustness and generalizability.

Framework: The study utilizes Head-Driven Phrase Structure Grammar (HPSG) and the English Resource Grammar (ERG). Unlike statistical parsers or Universal Dependencies, the ERG is a deterministic, exhaustive chart parser designed to provide structure for all well-formed English sentences, allowing for the analysis of rare linguistic phenomena and parseability.
Datasets:
- Human Text: Lead paragraphs from The New York Times (NYT) from two distinct periods: 2023 (used in prior work) and 2025 (newly collected).
- LLM Text: Two generations of models were compared:
  - Older Generation (Pre-2023): Base models (LLaMA 7B/13B/30B/65B, Falcon 7B, Mistral 7B) trained prior to October 2023.
  - Newer Generation (2024): Instruction-tuned models (Qwen 2.5, Mistral 7B v0.3, GPT-4o, LLaMA 3.3) trained up to 2024.
- Generation Protocol: For the 2025 human data, LLMs were prompted with NYT headlines and the first three words of the lead paragraph to generate synthetic leads. Instruction-tuned models required explicit system/user prompts to simulate the completion task.
Metrics: Diversity was quantified using ecological and information theory indices:
- Shannon-Wiener Index ( $H'$ ): Measures richness and evenness of distribution.
- Simpson Index ( $D$ ): Measures the probability that two randomly selected tokens belong to different categories.
- These metrics were applied to syntactic types (phrase structure configurations) and lexical types (complex HPSG representations of word constraints, distinct from surface vocabulary).

Key Contributions

Diachronic Comparison: Unlike prior work comparing a single generation of LLMs to humans, this study directly contrasts two generations of LLMs (base vs. instruction-tuned) against two temporal samples of human writing.
Formal Linguistic Analysis: The use of HPSG/ERG provides a linguistically grounded, theory-independent method for analyzing syntax and lexicon, avoiding biases inherent in NLP task performance or coarse Part-of-Speech tagging.
Diversity Metrics Application: The application of Shannon and Simpson indices to grammatical structures offers a systematic way to track stylistic shifts across model generations.

Results

Human Stability: English news text from 2023 and 2025 showed remarkable stability in both syntactic and lexical diversity metrics, indicating a consistent balance of variety in professional journalism.
LLM Divergence:
- Syntactic Diversity: Newer instruction-tuned models (2025) displayed significantly reduced syntactic diversity compared to older base models (2023). While older models were closer to human diversity, newer models favored constructions involving modification and coordination (e.g., noun phrases with modifiers) while avoiding specific details like names and dates.
- Lexical Diversity: The decline was even more pronounced in lexical diversity. Surprisingly, older LLMs (2023) exhibited the highest lexical diversity, surpassing both human text and newer models. Newer models (2025) showed the lowest lexical variety, relying on "basic" lexical types (adjectives, common nouns, transitive verbs) and avoiding specific referents.
- Punctuation: Newer models used punctuation with significantly less diversity, generating fewer em-dashes, semicolons, and quotations compared to human writers and older models.
Parsability and Length:
- Newer models generated longer sentences (15–30% longer than human text) compared to older models (which were shorter than human text).
- Despite increased length, newer LLM outputs were easier to parse (approaching 99% success rate) and required fewer computational resources than human text or older LLM outputs. This suggests that newer models produce structurally regular, streamlined sentences that align with the most predictable parts of the grammar, whereas human text triggers more complex, less predictable phenomena.

Significance and Claims
The paper claims that while instruction tuning enhances coherence and adherence to prompts, it has the side effect of narrowing the expressive range of model output. The findings suggest a trade-off where alignment paradigms sacrifice lexical and syntactic variety for stylistic control and factual safety (evidenced by the avoidance of specific names/dates and reduced hallucination-prone structures).

The authors conclude that larger, newer models are not converging toward human linguistic behavior in professional news writing; rather, they are diverging, becoming more formulaic and less diverse than both humans and earlier base models. This highlights the need for future research into how instruction tuning affects linguistic capabilities and whether these capabilities can be restored without compromising alignment. The study is modestly scoped to English news text, noting that the methodology is broadly applicable but currently limited by the availability of wide-coverage HPSG grammars for other languages.