ltzGLUE: Luxembourgish General Language Understanding Evaluation

1. Problem Statement

Despite the rapid advancement of Large Language Models (LLMs) and the proliferation of Natural Language Understanding (NLU) benchmarks for major European languages, Luxembourgish (LTZ) remains significantly under-researched. As an official national language with approximately 400,000 speakers, LTZ suffers from:

• Lack of Unified Benchmarks: Existing NLU tasks are fragmented, often limited to the news domain, and typically contain fewer than 1,000 instances, making model evaluation unreliable.
• Data Scarcity: There is a shortage of high-quality, labeled datasets for diverse NLP tasks.
• Linguistic Complexity: The language faces challenges related to ongoing standardization, vast orthographic variation, and decentralized resources.
• Evaluation Gap: There is no systematic framework to assess whether multilingual models or specialized LLMs truly understand Luxembourgish, leading to assumptions of capability that may not hold in practice.

2. Methodology

The authors developed LTZGLUE, the first unified GLUE-style benchmark for Luxembourgish, comprising 8 distinct NLU tasks. The methodology involved a hybrid approach of reusing existing resources and creating new datasets to ensure coverage across binary classification, multi-class classification, and sequence labeling.

2.1 Dataset Construction

The data sources primarily include RTL3 (the main Luxembourgish news broadcaster, 2008–2024) and Wikipedia (approx. 66k articles).

• Headline Acceptability (HA): A binary task determining if a headline matches an article body. Negative examples were created by swapping headlines between semantically similar but temporally distinct articles using TF-IDF cosine similarity.
• Sentiment Analysis (SA): A 3-class task (Positive, Negative, Neutral) derived from RTL commentary and letters to the editor, annotated by native speakers.
• Linguistic Acceptability (LA): A dataset derived from the Luxembourgish Online Dictionary (LOD) and spellcheckers. It includes four error types: Verb agreement, Adjective agreement, Syntax (deletion), and Orthography. It supports both binary and multi-class classification.
• Named Entity Recognition (NER): A merged corpus combining the automatically constructed JUDGEWEL (Wikipedia/Wikidata) and the human-annotated Lothritz et al. (RTL comments) datasets. It covers PER, ORG, LOC, DATE, and MISC entities.
• Topic Classification (TC): A 5-class task (Sports, Culture, Technology, Business, Animals) based on RTL news categories, with downsampling applied to address class imbalance.
• Intent Detection (ID): A translation of the English xSID dataset into Luxembourgish, focusing on voice-assistant commands (e.g., setting alarms, booking restaurants).
• Recognizing Textual Entailment (RTE): A machine-translated version of the original RTE dataset, significantly improved via LLM-assisted refinement (using ChatGPT-5) and manual verification to correct logical contradictions and label errors.

2.2 Model Evaluation

The authors evaluated two categories of models:

1. Supervised Encoder-Based Models:
   • LTZ-E1 (Mini/Base): Two new encoder models (68M and 110M parameters) pre-trained specifically on Luxembourgish using the "Ettin" recipe (based on ModernBERT).
   • Existing Models: LUXEMBERT (LTZ-specific), MBERT, MMBERT (updated multilingual BERT), and XLM-RoBERTa.
2. Unsupervised Prompted LLMs:
   • Zero-shot evaluation of GPT-5-Mini, Qwen3-235B, Llama-3.3, and Gemma-3-27B using prompt engineering without fine-tuning.

3. Key Contributions

1. LTZGLUE Benchmark: The first comprehensive, unified evaluation suite for Luxembourgish, featuring 8 diverse tasks that mirror the scope of the original English GLUE benchmark.
2. LTZ-E1 Models: The release of two new pre-trained encoder models (Mini and Base) specifically for Luxembourgish, which achieve competitive performance when fine-tuned.
3. Systematic Evaluation: A rigorous comparison of supervised fine-tuning versus zero-shot prompting, providing empirical evidence on the capabilities of current models for low-resource languages.
4. Resource Release: Public availability of the datasets, code, and model checkpoints to foster further research in Luxembourgish NLP.

4. Results

The evaluation (reported as Macro-F1 scores) reveals distinct performance patterns:

• Supervised Encoders vs. Prompted LLMs:
  • Fine-tuned encoders consistently outperform prompted LLMs, particularly on tasks requiring structured prediction or fine-grained linguistic discrimination (e.g., NER, Intent Detection, Linguistic Acceptability).
  • Prompted LLMs show high variability and generally underperform, except on semantically coarse-grained tasks like Topic Classification (where they approach encoder performance) and Textual Entailment (where GPT-5 and Qwen perform surprisingly well, rivaling some encoders).
• Best Performing Models:
  • MMBERT-base (Multilingual BERT with updated data) emerged as the most robust model overall, achieving the highest or near-highest scores across most tasks with low variance.
  • LUXEMBERT and LTZ-E1-mini showed strong performance on lexically grounded tasks (e.g., Intent Detection, Acceptability) but exhibited higher instability on complex inference tasks.
  • LTZ-E1-base and XLM-R generally lagged behind in consistency and performance.
• Task Difficulty:
  • Easiest: Topic Classification (high scores across all models).
  • Hardest: Recognizing Textual Entailment (RTE) and Intent Detection (ID), where even top models struggle, and prompted LLMs often fail significantly.

5. Significance and Conclusion

• Paradigm Shift for Low-Resource Languages: The paper demonstrates that while LLMs are powerful, task-specific fine-tuning of encoder models remains the superior approach for structured NLU tasks in low-resource languages like Luxembourgish. Prompting alone is insufficient for reliable performance in these settings.
• Foundation for Future Research: LTZGLUE provides a necessary baseline for the Luxembourgish NLP community, moving beyond fragmented, small-scale evaluations to a standardized, reproducible framework.
• Resource-Conscious Strategy: The authors highlight a methodology for building benchmarks in data-scarce environments by combining existing resources, targeted annotation, and LLM-assisted data cleaning, offering a blueprint for other under-researched languages.
• Limitations: The authors acknowledge that the dataset is constrained by the availability of source material (mostly formal/news text), potentially limiting ecological validity for informal or sociolinguistically diverse contexts. Additionally, potential data leakage (pre-training data overlap) is noted as a factor that may inflate LLM performance estimates.

In summary, LTZGLUE establishes a critical infrastructure for Luxembourgish NLP, proving that while generalist LLMs have potential, specialized, fine-tuned encoders are currently the most reliable solution for understanding the Luxembourgish language.