💬 NLP

Measuring Grammatical Diversity from Small Corpora: Derivational Entropy Rates, Mean Length of Utterances, and Annotation Invariance

This paper proposes a theory-free framework for measuring grammatical diversity from small corpora by establishing a fundamental link between derivational entropy and mean length of utterance (MLU), introducing the derivational entropy rate and the Smoothed Induced Treebank Entropy (SITE) tool to accurately assess syntactic complexity across different annotation frameworks.

Original authors: Fermin Moscoso del Prado Martin

Published 2026-06-04

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Fermin Moscoso del Prado Martin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Counting the "Flavor" of Language

Imagine you are a food critic trying to judge the variety of a chef's cooking. You have a small sample of their dishes (a "corpus"). You want to know: How diverse is this chef's menu? Do they only make plain pasta, or do they have a vast, complex repertoire of sauces, spices, and techniques?

In linguistics, researchers face the same problem. They want to measure the grammatical diversity (or "syntactic complexity") of a person or a group. Do they use simple sentences, or do they weave together complex, nested structures?

The problem is that we usually only have a tiny sample of what someone says (like a few pages of a diary or a short conversation). Traditional methods for measuring diversity often fail with small samples because they get confused by the limited data.

This paper introduces a new way to measure this diversity that works even with very small samples, and it reveals a surprising secret: The length of a sentence is actually a direct measure of its complexity.

The Old Way vs. The New Way

The "Proxy" Problem

For a long time, researchers used a simple trick called Mean Length of Utterance (MLU). This is just counting the average number of words in a sentence.

The Analogy: Imagine judging a chef's skill by the size of the plate. "Wow, this plate is huge! They must be using complex techniques!"
The Flaw: Sometimes a huge plate is just a giant salad (simple), and a tiny plate is a delicate, multi-layered soufflé (complex). Researchers thought MLU was just a "proxy"—a guess that usually worked but wasn't the real thing.

The "Entropy" Problem

To get the real measure of complexity, linguists use a concept called Derivational Entropy.

The Analogy: Imagine a "Grammar Tree." Every time you speak, you are growing a branch on this tree. Entropy measures how many different ways that tree could have grown. High entropy means the tree could have grown in millions of different directions (high diversity). Low entropy means it only grows in a few straight lines (low diversity).
The Flaw: Calculating this requires a massive amount of data. If you only have a few sentences, the math gets messy and biased. It's like trying to guess the entire menu of a restaurant after seeing only one dish; you might think they only serve that one dish, missing the rest of the menu.

The Big Discovery: The "Entropy Rate"

The author of this paper discovered a fundamental link between the two concepts above. He found that Sentence Length (MLU) and Grammatical Diversity (Entropy) are not just related; they are mathematically locked together.

He calls this new measure the Derivational Entropy Rate.

The Analogy: Imagine a "Complexity Tax."
- Every time you add a word to a sentence, you are paying a tax.
- The Derivational Entropy Rate is the price of that tax.
- The author found that for a specific type of language (like American English) and a specific way of analyzing it (like a specific grammar rulebook), this "price" is constant.
- If you know the average length of the sentences, you can calculate the exact diversity. If you know the diversity, you can calculate the length. They are two sides of the same coin.

Why is this huge?
It means the "simple" measure (counting words) is actually a perfect measure of complexity, provided you know the "tax rate" (the entropy rate) for that specific language and analysis style. You don't need complex math to guess the diversity; you just need to count the words.

The Tool: "SITE" (The Smoothing Sponge)

The paper also introduces a tool called SITE (Smoothed Induced Treebank Entropy).

The Problem: When you have a tiny sample (like 50 sentences), standard math tools get "biased." They think the diversity is lower than it really is because they haven't seen all the possible sentence types yet.
The Solution: SITE is like a smart sponge. It takes the small, dry sample of data and "soaks" it with statistical corrections. It fills in the gaps of what could have been said but wasn't.
The Result: SITE can accurately guess the true diversity of a language using just 100 sentences (for standard grammar) or 1,000 sentences (for dependency grammar). Old methods needed 15,000+ sentences to get the same accuracy.

The "Annotation Invariance" Surprise

The author tested this on two different ways of analyzing language:

Constituency Grammar (CFG): Like looking at a sentence as a set of nested boxes (Noun Phrase inside a Verb Phrase).
Dependency Grammar (DG): Like looking at a sentence as a web of connections between words (Word A connects to Word B).

The Finding:
Even though these two methods look at sentences very differently, the Derivational Entropy Rate remained constant within each method.

If you use Method A, the "tax rate" is $X$ .
If you use Method B, the "tax rate" is $Y$ .
But once you pick a method, the rate stays the same no matter who is speaking or what they are talking about.

This means that if you compare two different groups of people (e.g., children vs. adults), you can trust the results as long as you use the same analysis method for both. The method itself doesn't change the relative ranking of who is more complex.

What About "Messy" Data?

The author tested this on a huge, messy historical corpus (IcePaHC) containing texts from the 12th century to the 21st century.

The Result: When you mix all these different eras and authors together, the math breaks. The "diversity" number keeps bouncing up and down and never settles.
The Lesson: This tells us that the "Grammar" of a language isn't one single, static thing. It changes over time and between authors. The tool (SITE) is smart enough to tell you: "Hey, this data is too mixed up to give you a single number."
However, if you look at one single author at one specific time, the math works perfectly, and the diversity converges quickly.

Summary of Claims

Length = Diversity: The average length of a sentence is not just a guess; it is a fundamental, theoretical measure of grammatical diversity.
The Rate is Constant: For a specific language and a specific way of analyzing it, the "complexity per word" (Entropy Rate) is a fixed constant.
Small Samples Work: Using the new SITE method, you can get accurate diversity measurements from very small samples (as few as 100 sentences), whereas old methods needed massive datasets.
Method Matters: If you change how you analyze the grammar (e.g., from boxes to webs), the numbers change, but the relationship between length and diversity stays consistent within that method.
Homogeneity is Key: These measurements only work if the data comes from a consistent source (one speaker, one time period). If you mix everything together, the measurement fails to converge, which is actually a useful signal that the data is too diverse to be treated as one single group.

In short: You don't need a supercomputer to measure how complex someone's language is. If you count their words and know the "rules of the game" (the annotation style), you have the answer. And if you have a tiny sample, use the new "SITE" sponge to get the right answer quickly.

Technical Summary: Measuring Grammatical Diversity from Small Corpora

Problem Statement
In fields ranging from language acquisition and neuropsychology to historical linguistics, researchers frequently rely on small corpora to estimate the grammatical diversity (syntactic complexity) of individuals or communities. A primary challenge in this domain is the bias inherent in estimating the entropy of Probabilistic Context-Free Grammars (PCFGs) induced from limited data. Standard Maximum-Likelihood (ML) estimators are known to severely underestimate true entropy, a problem exacerbated by the potentially infinite number of syntactic structures a grammar can generate and the Zipfian distribution of these structures. Consequently, researchers often resort to "proxy" measures like the Mean Length of Utterance (MLU) or specific construction counts, which are either theory-bound or fail to capture the full diversity of syntactic structures. Furthermore, there is a lack of understanding regarding how different grammatical annotation frameworks (e.g., constituency vs. dependency) affect the comparability of diversity measures across corpora.

Methodology
The paper proposes a theoretical and empirical framework linking MLU to derivational entropy through the concept of the derivational entropy rate.

Theoretical Link: The author establishes that within a PCFG, the derivational entropy ( $H[G]$ ) and the mean length of utterance ($MLU[G]$) are fundamentally linked. By defining the derivational entropy rate ( $h[G]$ ) as the ratio $H[G]/MLU[G]$, the paper demonstrates that these two measures are directly proportional. This relationship implies that MLU is not merely a proxy but a fundamental, theory-free measure of syntactic diversity, differing from entropy only by a constant scaling factor (the rate).
SITE Estimator: To address the bias in small corpora, the paper utilizes and refines the Smoothed Induced Treebank Entropy (SITE) method. This approach involves:
- Inducing a PCFG from a treebank using Maximum Likelihood.
- Decomposing the entropy estimation into local expansion entropies for each non-terminal symbol.
- Applying bias-correction estimators (specifically the CWJ estimator by Chao, Wang, and Jost, 2013) to the local expansion entropies while leaving the characteristic matrix (structural counts) uncorrected, as the latter converges rapidly and is unbiased.
- Extending this method to Dependency Treebanks by converting dependency graphs into equivalent context-free derivation trees, preserving all structural information.
Empirical Validation: The methodology is tested on:
- Penn Treebank (WSJ subsample): Both context-free and dependency versions, where the "true" entropy of the induced grammar is known, allowing for convergence analysis.
- IcePaHC (Icelandic Parsed Historical Corpus): A large, heterogeneous, diachronic corpus with both context-free and Universal Dependencies (UD) annotations, used to test the stability of the derivational entropy rate across different languages, time periods, and annotation schemes.

Key Contributions

Derivational Entropy Rate: The introduction of a new metric, the derivational entropy rate ( $h[G]$ ), which quantifies the rate at which grammatical diversity grows per added symbol.
Re-evaluation of MLU: Theoretical proof and empirical demonstration that MLU is a fundamental measure of syntactic diversity, mathematically equivalent to derivational entropy within a specific annotation framework.
SITE Method: A robust, bias-corrected estimator for derivational entropy that converges significantly faster (requiring only ~100 sentences for context-free and ~1,000 for dependency grammars) than traditional ML or Monte Carlo methods.
Annotation Invariance: Evidence that while absolute entropy values depend on the annotation scheme (e.g., CFG vs. DG), the relative diversity of texts remains invariant. If two corpora are annotated with the same convention, their entropy rates are constant, allowing for cross-corpus comparisons.

Results

Convergence: The SITE method with CWJ smoothing achieves unbiased convergence to true entropy values with very small sample sizes, whereas ML methods require orders of magnitude more data and remain biased even with large samples.
Linearity: Strong linear correlations (Pearson's $r > 0.98$ after bias correction) were found between MLU and derivational entropy across various subcorpora, confirming the theoretical proportionality. The regression lines pass through the origin, indicating no significant intercept.
Entropy Rate Constancy: The derivational entropy rate was found to be constant within a specific annotation convention across diverse texts (e.g., within the IcePaHC corpus and the Penn Treebank). However, the rate differs between annotation conventions (e.g., CFG vs. DG), falsifying the hypothesis that the rate is constant across all languages regardless of annotation, but supporting the hypothesis that it is constant within a language and annotation scheme.
Heterogeneity: In highly heterogeneous corpora (mixing different authors, genres, and historical periods), entropy estimates fail to converge, even with randomization. This suggests that SITE can serve as a diagnostic tool for assessing the homogeneity of a corpus.

Significance and Claims
The paper claims that these results provide a "theory-free assessment of grammatical complexity." By validating MLU as a direct measure of syntactic diversity (scaled by the annotation-specific entropy rate), researchers can bypass complex entropy calculations for many applications, relying instead on the simple, unbiased, and fast-converging MLU.

The derivational entropy rate is presented as a crucial complement for cases where explicit entropy values are required (e.g., estimating parsing difficulty or grammar ambiguity). The findings suggest that syntactic diversity is an inherent property of linguistic samples that is independent of the specific theoretical representation (CFG or DG), provided the annotation scheme is consistent. This allows for the comparison of grammatical complexity across different languages or historical periods, provided that comparable annotation conventions are used. The work also highlights the limitations of applying global entropy measures to non-stationary, heterogeneous data, offering a method to detect such heterogeneity.