Community size rather than grammatical complexity better predicts Large Language Model accuracy in a novel Wug Test

1. Problem Statement

The linguistic capabilities of Large Language Models (LLMs) remain a subject of debate, particularly regarding whether they possess genuine grammatical competence or merely mimic human performance through statistical pattern matching. While previous studies have investigated LLMs in syntactic and semantic domains, there is limited understanding of their morphological generalization abilities (the ability to apply rules to novel words).

Existing literature presents conflicting findings:

• Some studies suggest LLM performance is driven by linguistic complexity (e.g., models perform better on simpler languages).
• Others suggest performance is driven by data availability (e.g., models perform better on languages with larger speaker communities and more training data).

The authors aim to resolve this ambiguity by systematically testing whether LLM accuracy in morphological tasks is primarily determined by the structural complexity of a language or the size of its speaker community (as a proxy for training data volume).

2. Methodology

Experimental Design: Multilingual Wug Test

The study utilized a modified version of the classic Wug Test, designed to assess the ability to apply grammatical rules to nonce (invented) words.

• Task: Participants (humans and LLMs) were presented with sentences containing a singular nonce word (e.g., "Yesterday a glorp appeared...") and asked to provide the plural form (e.g., "Now there are two glorps").
• Stimuli: 30 nonce words per language (15 two-syllable, 15 three-syllable). Words were constructed to respect the morphophonological constraints of the target languages, specifically altering initial consonants to ensure robust lexical identification.
• Languages Tested: Four partially unrelated languages with varying complexity and community sizes:
  1. English (Low complexity, ~1.5B speakers)
  2. Spanish (Medium complexity, ~488M speakers)
  3. Greek (High complexity, ~12M speakers)
  4. Catalan (High complexity, ~8M speakers)

Participants and Models

• Human Participants: 160 native speakers (40 per language) recruited via Prolific.
• LLMs Tested: Six models were evaluated: ChatGPT-3.5, ChatGPT-4, Grok 3, BERT, DeepSeek, and Mistral.
  • Note: BERT was evaluated programmatically via Python; others were tested manually via user interfaces.

Complexity Metrics

To quantify linguistic complexity, the authors used Grambank v1.0 to calculate two dimensions for each language:

1. Fusion: The extent of phonologically bound markers (affixes).
2. Informativity: The number of obligatory grammatical distinctions.

• Resulting Complexity Ranking: English < Spanish < Catalan < Greek.

Data Analysis

• Metric: Accuracy (binary: correct/incorrect) based on adherence to morphophonological rules.
• Statistical Approach: Mixed-effects logistic regression models (glmer in R) were used to analyze the effects of Agent (Human vs. LLM), Language, Complexity Score, and Community Size. Model selection was performed using Akaike Information Criterion (AIC).

3. Key Contributions

1. Systematic Cross-Linguistic Comparison: Unlike previous studies that focused on single models or limited language sets, this study compares six diverse LLMs across four languages with distinct typological profiles.
2. Disentangling Complexity vs. Data: The study explicitly tests the competing hypotheses of whether LLMs are sensitive to grammatical structure or merely to the statistical frequency of data (community size).
3. Refinement of "Competence" Claims: The authors argue that while LLMs achieve human-like accuracy, the underlying drivers differ, suggesting a "superficial" resemblance to human linguistic competence.

4. Results

Human vs. Model Performance

• Overall Accuracy: LLMs (excluding BERT, which performed at floor level) achieved accuracy comparable to humans. In English and Spanish, top models (ChatGPT-4, DeepSeek, Mistral) even surpassed human accuracy (reaching 100% in Spanish vs. 95.7% for humans).
• Statistical Significance: When BERT was excluded, there was no statistically significant difference between human and LLM performance overall. However, a significant interaction was found between Agent and Language: humans performed significantly worse than LLMs in English, but performance was comparable in other languages.

The Predictors of Accuracy

The study tested two main hypotheses regarding what drives LLM performance:

1. Hypothesis A (Complexity): Performance should follow the complexity ranking (English > Spanish > Catalan > Greek).
2. Hypothesis B (Community Size): Performance should follow the speaker population ranking (English > Spanish > Greek > Catalan).

Findings:

• Community Size is the Stronger Predictor: Statistical modeling (comparing AIC scores) confirmed that Community Size is a significantly better predictor of accuracy than Complexity Score.
  • Observed Ranking: Spanish > English > Greek > Catalan.
  • This aligns with data availability (Spanish and English have massive digital corpora; Catalan and Greek are less represented).
• Complexity Interaction: While complexity had a significant effect, it was secondary. Interestingly, when community sizes were comparable, the model predicted that higher complexity correlated with higher accuracy (e.g., Greek outperformed Catalan despite being more complex), likely due to the regularity of Greek morphology compared to Catalan's orthographic irregularities.
• The English Anomaly: Despite having the largest community, English did not yield the highest model accuracy (Spanish did). The authors attribute this to the specific irregularity of the English stimuli used in the test, which caused both humans and models to overgeneralize rules (e.g., producing "sungi" instead of "sunguses").

5. Significance and Conclusion

• Resource Sensitivity over Structural Sensitivity: The primary conclusion is that LLM behavior is driven by the richness of linguistic resources (training data volume) rather than an intrinsic sensitivity to grammatical complexity. Models excel in languages with abundant digital representation, regardless of structural simplicity.
• Nature of LLM "Competence": The findings suggest that LLMs replicate human performance superficially. They generalize rules effectively in high-resource environments but do so through distributional learning (tokenization and pattern matching) rather than deep structural understanding.
• Implications for Low-Resource Languages: The study highlights a performance gap for languages with smaller speaker communities (like Catalan), even when those languages are structurally similar to high-performing ones (like Spanish). This underscores the "digital divide" in AI capabilities.
• Future Directions: The authors call for more dedicated metrics of morphological complexity (isolating it from syntax/semantics) and testing on a broader range of typologically diverse languages (e.g., agglutinative or polysynthetic) to further validate these findings.

In summary, the paper demonstrates that while LLMs can master morphological generalization tasks to a human-like degree, their success is a function of data availability (community size) rather than grammatical complexity, challenging the notion that they possess a deep, structure-based understanding of language.