Evaluating Metalinguistic Knowledge in Large Language Models across the World's Languages

1. Problem Statement

While Large Language Models (LLMs) are routinely evaluated on language use (generation, translation, QA), their explicit knowledge of language structure (metalinguistic knowledge) remains poorly understood. Existing benchmarks suffer from three critical limitations:

• Narrow Scope: They focus on specific linguistic phenomena (e.g., agreement, acceptability) rather than broad structural features.
• Resource Bias: They heavily favor high-resource languages (especially English), neglecting the vast majority of the world's languages.
• Lack of Generalizability: There is no systematic understanding of how LLMs handle metalinguistic reasoning across the full spectrum of human linguistic diversity, particularly for low-resource and under-documented languages.

This gap is critical because LLMs are increasingly used for language documentation and preservation; if they lack explicit grammatical knowledge, their utility in these fields is compromised.

2. Methodology

Benchmark Construction (WALS-based)

The authors constructed a massive multilingual benchmark based on the World Atlas of Language Structures (WALS), a database containing 192 structural features across 2,660 languages.

• Transformation: WALS features were converted into natural language multiple-choice questions (MCQs). For example, a feature regarding "Nominal Plurality" was transformed into a question asking how plurality is marked in a specific language, with options reflecting the possible WALS values.
• Refinement: Complex or terminology-heavy WALS labels were rephrased into clear, interpretable questions to minimize surface-level memorization and ensure the model must reason about the structure.
• Dataset: The final benchmark comprises 192 distinct question types instantiated for every language-feature pair where data exists, resulting in a dataset covering 2,660 languages.

Experimental Setup

• Models Evaluated:
  • GPT-4o: A state-of-the-art proprietary model.
  • Llama-3.3-70B: A large open-source model.
  • Gemma-3-27B: A medium-sized open-source model.
• Prompting Strategy: Zero-shot prompting with a fixed temperature (0.2) to ensure deterministic outputs. Models were asked to select one option from a predefined list without explanation.
• Evaluation Metrics:
  • Accuracy: Proportion of correct answers.
  • Macro F1: To account for class imbalance in feature values.
  • Baselines:
    • Chance Baseline: Random selection among options.
    • Majority-Class Baseline: Always predicting the most frequent value for a feature.

Predictive Analysis

To understand performance variations, the authors analyzed correlations between model accuracy and external factors:

• Domain Level: Online visibility (Google search hit counts) of specific linguistic features.
• Language Level: Eight predictors including digital status (Joshi et al. taxonomy), Wikipedia size, Universal Dependencies (UD) corpus size, language vitality, geographical macroregion, language family, and typological distance to English (via Lang2vec).

3. Key Contributions

1. New Multilingual Benchmark: Introduction of the first massively multilingual benchmark for metalinguistic knowledge, covering 2,660 languages and 192 grammatical features, released as an open-source dataset.
2. Large-Scale Evaluation: A systematic evaluation of LLMs across the full spectrum of documented languages, including a significant proportion of low-resource and under-documented languages.
3. Methodological Insights: A critical analysis of using typological databases (like WALS) for benchmarking, highlighting issues of sparsity, categorical design, and the distinction between surface patterns and deep grammatical competence.

4. Key Results

Overall Performance

• Limited Knowledge: All models performed poorly. GPT-4o achieved the highest accuracy (0.367), followed by Llama-3.3-70B (0.265) and Gemma-3-27B (0.246).
• Baseline Failure: While all models significantly outperformed the chance baseline (0.234), none outperformed the majority-class baseline (0.539). This indicates models capture broad cross-linguistic patterns (e.g., "most languages use suffixes for plural") but fail to distinguish fine-grained, language-specific grammatical distinctions.

Performance by Linguistic Domain

• Variation: Performance varied significantly by domain.
  • Highest Accuracy: Lexicon and Verbal categories.
  • Lowest Accuracy: Phonology, Nominal Syntax, and Sign Languages.
• Correlation with Visibility: For GPT-4o, accuracy strongly correlated with the online visibility of the linguistic feature (Google search hits, $r=0.715$). This suggests models rely on the frequency of these concepts in their training data rather than abstract grammatical reasoning.

Performance by Language

• Digital Divide: There is a strong, consistent association between a language's digital status and model accuracy. High-resource languages (e.g., English, Spanish, Mandarin) achieved high accuracy, while low-resource languages (e.g., Barasano, Imonda) performed near chance levels.
• Predictive Factors: A Random Forest classifier identified Resource Availability (Wikipedia size, UD corpus size) as the strongest predictors of accuracy.
  • Typological Proximity: Distance to English was the third strongest predictor, suggesting models leverage structural similarities to English.
  • Weak Predictors: Genealogical family and geographical macroregion were the weakest predictors, indicating that being in the same language family as English does not guarantee performance if digital resources are scarce.

5. Significance and Implications

• Fragmented Competence: The study concludes that LLMs do not possess a generalizable "grammatical competence." Instead, their metalinguistic knowledge is a fragmented reflection of the distribution of data in their training corpora.
• Bias Reinforcement: The results highlight a critical risk: LLMs appear linguistically competent only for digitally dominant languages, potentially reinforcing biases in linguistic research and language documentation efforts.
• Future Directions: The paper argues for the necessity of inclusive evaluation frameworks that explicitly test low-resource languages. It suggests that future models must be trained on more diverse, curated linguistic data rather than relying solely on raw internet text to achieve true metalinguistic understanding.
• Resource Availability: By releasing the benchmark, the authors provide a tool for the community to systematically track progress in global linguistic diversity and the development of more robust, equitable language models.