Targeted Syntactic Evaluation of Language Models on Georgian Case Alignment

1. Problem Statement

The paper addresses the lack of syntactic evaluation benchmarks for Low-Resource Languages (LRLs), specifically focusing on Georgian. While Targeted Syntactic Evaluation (TSE) using minimal pairs is a standard method for assessing the grammatical understanding of Large Language Models (LLMs) in high-resource languages (like English), such resources are scarce for Georgian.

The specific linguistic phenomenon under investigation is split-ergative case alignment. Georgian exhibits a complex system where grammatical cases (Nominative, Ergative, Dative) assigned to subjects and objects change based on verb tense, aspect, and mood (referred to as screeves).

• The Challenge: Current models struggle with low-resource languages, and it is unclear if they have truly learned the underlying syntactic rules of split-ergativity or if they are merely memorizing surface patterns. The paper aims to quantify how well transformer-based models understand these specific case-alignment rules.

2. Methodology

The authors propose a treebank-based approach to generate minimal pairs for syntactic testing, utilizing the Georgian Language Corpus Universal Dependencies (GLC UD) treebank and the Grew graph-based query language.

Data Generation

• Source: 3,013 annotated sentences from the GLC UD treebank.
• Query Language: Grew was used to extract specific syntactic patterns corresponding to four construction types:
  1. Intransitive Nominative (Accusative alignment).
  2. Transitive Nominative–Dative (Accusative alignment).
  3. Transitive Ergative–Nominative (Ergative alignment).
  4. Transitive Dative–Nominative (Inverted alignment).
• Minimal Sets: Unlike standard binary minimal pairs (grammatical vs. ungrammatical), the authors created minimal sets of three. For each test sentence, the model is presented with a target noun in three forms: Nominative, Ergative, and Dative. The model must assign the highest probability to the grammatically correct form for that specific context.
• Dataset Size: 370 total tests across 7 distinct tasks (50–70 samples per task), covering both subject and object roles in transitive and intransitive constructions.

Models Evaluated

Seven models were evaluated, ranging from 110M to 1.3B parameters:

• Encoder-only: XLM-RoBERTa (Base/Large), mBERT, RemBERT, and HPLT-BERT-ka (Georgian-only).
• Decoder-only: GPT2-geo and mGPT-ka.
• Metrics:
  • Word-Level (WL) Accuracy: Measures the probability of the target token given the context before the token is generated.
  • Sentence-Level (SL) Accuracy: Measures the joint probability of the entire sentence.
  • Note: WL is preferred for encoder models, while SL is necessary for decoder-only models to account for full context visibility.

3. Key Contributions

1. First Georgian Syntactic Benchmark: The creation and public release of the first dedicated dataset (370 tests) for evaluating Georgian case alignment, addressing a significant gap in LRL research.
2. Query-Based Generation Framework: Demonstrated a generalizable methodology using Grew and Universal Dependencies treebanks to generate minimal pairs for languages where standard benchmarks do not exist.
3. Granular Analysis of Split-Ergativity: Provided a detailed breakdown of model performance across the three specific case alignments (Nominative, Ergative, Dative) rather than treating syntax as a monolithic block.

4. Results

The evaluation revealed distinct performance patterns correlated with the frequency of case usage in the training data.

• Performance Hierarchy: Models performed best on the Nominative case, followed by the Dative, and worst on the Ergative case.
  • Nominative: Avg Word-Level Accuracy ~88.6%.
  • Dative: Avg Word-Level Accuracy ~48.8%.
  • Ergative: Avg Word-Level Accuracy ~24.4%.
• The Ergative Deficit: Models struggled significantly to assign the Ergative case correctly. This case is used in specific "past tense" (Series II) contexts. The low performance suggests models fail to learn the specific triggers for ergativity, likely due to data scarcity (Ergative appears ~24x less frequently than Nominative in the treebank).
• Nominative Bias: When models failed to predict the correct case, they exhibited a strong bias toward defaulting to the Nominative case.
  • When Dative was correct but predicted wrong, 83.2% of errors were Nominative.
  • When Ergative was correct but predicted wrong, 74.4% of errors were Nominative.
• Model Comparison: The mGPT-ka (1.3B parameters, fine-tuned on Georgian) achieved the highest overall accuracy, particularly on the difficult Ergative tasks, suggesting that larger, language-specific fine-tuning helps mitigate data scarcity.
• Tokenization Impact: Pearson correlation analysis showed no significant correlation between token/sentence length and performance, ruling out tokenization complexity as the primary cause of errors.

5. Significance and Conclusion

• Data Scarcity vs. Syntactic Complexity: The study isolates the impact of data scarcity on syntactic learning. It concludes that the poor performance on the Ergative case is not just a general LRL issue but is exacerbated by the rarity of the specific syntactic feature (split-ergativity) in the training corpus.
• Generalizability: The Grew-based methodology offers a scalable solution for creating syntactic benchmarks for other low-resource languages that have existing treebanks but lack standardized evaluation suites.
• Future Work: The authors suggest expanding the dataset to include more screeves (verb forms) and intransitive tests to further probe the limits of model generalization in Georgian and similar languages.

In summary, the paper demonstrates that while transformer models can capture common syntactic patterns (Nominative) in low-resource languages, they struggle significantly with rare, complex morphological features (Ergative) when training data is insufficient, often reverting to a "default" case rather than applying learned grammatical rules.