Do LLMs Know What Luxembourgish Borrows? Probing Lexical Neology in Low-Resource Multilingual Models

Technical Summary: Do LLMs Know What Luxembourgish Borrows? Probing Lexical Neology in Low-Resource Multilingual Models

Problem Statement
Large Language Models (LLMs) are increasingly deployed for writing assistance in small contact languages like Luxembourgish. However, it remains unclear whether these models respect community norms regarding lexical borrowing and neology (the creation of new words). Luxembourgish exists in a dense contact zone with German, French, and English, where modern lexical growth relies heavily on morphologically and orthographically adapted borrowings rather than endogenous coinages. Current LLMs, trained on massive multilingual corpora, may fail to distinguish between legitimate integrated loanwords, code-switching, and native vocabulary, potentially treating adapted forms as errors or normalizing them back to donor languages. This study investigates whether off-the-shelf multilingual LLMs can correctly identify native Luxembourgish words versus borrowings from French, German, or English, and whether they can detect lexical innovation.

Methodology
The authors introduce LexNeo-Bench, a token-level benchmark derived from LuxBorrow, a large-scale corpus of professionally edited Luxembourgish news (RTL, 1999–2025). The benchmark consists of 3,050 instances where target tokens are labeled as NATIVE, FR_LOAN (French), DE_LOAN (German), or EN_LOAN (English). Note that EN_LOAN is used as a diagnostic distractor in prompts but is excluded from the evaluation set due to low frequency.

The study evaluates three instruction-tuned multilingual LLMs (Gemma 3 12B, Gemma 3 27B, and Llama 3.3 70B) in a frozen, prompt-only setting across 34 configurations. The evaluation focuses on two tasks:

1. Borrowing Classification: A four-way classification task (NATIVE, FR_LOAN, DE_LOAN, EN_LOAN).
2. Neology Detection: A binary task determining if a token represents a lexical innovation (borrowing) or a native item.

To address the limitations of parametric knowledge, the authors construct a Linguistic Knowledge Graph (LKG) integrating morphological patterns, a loanword lexicon from the Lëtzebuerger Online Dictionnaire (LOD), and synonym pairs. They compare several prompting strategies:

• Zero-shot: Standard instructions without context.
• Few-shot: Instructions with five manually curated examples.
• KG_flat: A global list of 19 productive morphological patterns appended to every prompt.
• KG_graph: An instance-specific subgraph retrieved from the LKG, containing token-specific etymology, donor metadata, analogues, and contrastive patterns.

Key Contributions

1. LexNeo-Bench: The first token-level benchmark for borrowing classification in Luxembourgish, derived from a diachronic news corpus, with public scripts for extraction and evaluation.
2. Neology Proxy Task: The introduction of a binary lexical-innovation task to probe LLM awareness of neology, revealing that this task remains challenging even for strong models.
3. Lexicon-Aware Prompting: Demonstration that injecting structured, community-curated lexical resources (via knowledge graphs) significantly improves LLM performance in low-resource contact settings, offering a concrete avenue for integrating lexical resources into LLM workflows.

Results

• Baseline Performance: Without external context, LLMs perform only slightly above chance (24.5% – 34.7% accuracy) on borrowing classification, indicating that parametric knowledge alone is insufficient for distinguishing adapted borrowings in Luxembourgish.
• Impact of Knowledge Graphs: The KG_graph prompting strategy dramatically improves performance, raising accuracy to 71.3% – 81.0%. This improvement is attributed to instance-specific retrieval rather than the mere presence of global pattern lists (KG_flat performed similarly to non-KG baselines).
• Model Scale and Bias: Contrary to expectations, the smallest model (Gemma 3 12B) achieved the highest accuracy (81.0%) under KG_graph, outperforming the 27B and 70B models. Larger models exhibited a bias toward over-predicting German borrowings (DE_LOAN) and struggled to identify native words when graph evidence was incomplete, suggesting stronger internal priors for donor languages that conflict with contextual evidence.
• Donor Bias: Models showed a systematic bias toward German origins, misclassifying French borrowings as German at a rate 4.6 times higher than the reverse. They also frequently predicted EN_LOAN for native tokens in zero-shot settings, a bias suppressed by KG_graph prompting.
• Neology Detection: Unlike borrowing classification, the neology detection task did not benefit from KG_graph prompting; in fact, performance degraded. Few-shot prompting remained the most effective strategy for this task. The authors attribute this to the LKG encoding structural origin information (donor, morphology) rather than diachronic signals (recency, first attestation) required to define "neology."
• Binary vs. Multi-class: Distinguishing native words from any borrowing was easier (85.5% accuracy for 12B) than distinguishing the specific donor language, indicating that the primary difficulty lies in separating French from German borrowings in a dense contact zone.

Significance and Claims
The paper claims that for low-resource contact languages, lexicon-aware prompting is essential for robust borrowing judgments. The study demonstrates that structured linguistic context can partially compensate for sparse pre-training data, allowing even smaller models to outperform larger ones when provided with instance-specific evidence. However, the authors modestly note that while structural context aids in identifying where a word comes from, it does not solve diachronic questions regarding when a word entered the language, limiting its utility for neology detection without explicit temporal signals.

The work contributes to the ENEOLI COST Action by treating borrowing as a form of lexical innovation and providing a framework for evaluating how multilingual LLMs analyze such forms. The authors emphasize that these resources are descriptive tools for studying contact phenomena and stress-testing models, not prescriptive standards for "correct" language use. Limitations include the reliance on a single news source (under-representing informal registers) and the use of automatic annotation pipelines. Future work is proposed to extend the benchmark to other genres and integrate explicit diachronic signals.