Are Language Models Borrowing-Blind? A Multilingual Evaluation of Loanword Identification across 10 Languages

1. Problem Statement

The paper addresses the challenge of loanword identification—the task of distinguishing words borrowed from other languages (loanwords) from native vocabulary within a specific language. While lexical borrowing is a well-studied phenomenon in linguistics, it remains under-explored in Natural Language Processing (NLP).

The authors investigate whether modern Pretrained Language Models (PLMs) and Large Language Models (LLMs) possess the capability to identify loanwords when explicitly instructed. The study is motivated by the observation that NLP systems often exhibit a bias toward loanwords over native equivalents in generation and comprehension tasks. The core question is: If models prefer loanwords, can they accurately identify them when asked?

2. Methodology

The study utilizes the ConLoan dataset, a contrastive multilingual dataset containing sentence-level annotations where loanwords are paired with native alternatives across 10 typologically diverse languages (Chinese, French, German, Greek, Icelandic, Italian, Northern Kurdish, Portuguese, Russian, and Spanish).

The evaluation is conducted through two complementary approaches:

A. Prompting Large Language Models (LLMs)

Three LLMs were evaluated: Gemini-2.5-Flash-Lite, GPT-4.1, and Meta-Llama-3-8B-Instruct.

• Settings: Zero-shot and Few-shot (2 examples).
• Prompt Variants: Three prompts were tested to see how definition granularity affects performance:
  1. Minimal: Basic instruction without definition.
  2. Etymological: Defines loanwords based on historical borrowing (Haspelmath & Tadmor, 2009).
  3. Usage-based: Defines loanwords based on conventional use by monolinguals (distinguishing them from code-switching).
• Output Constraint: Models were constrained to output valid Python lists of strings for deterministic parsing.

B. Fine-tuning Multilingual Encoders

Four pretrained multilingual encoder models were evaluated: mBERT, XLM-RoBERTa (base and large), and ELECTRA-base.

• Configurations:
  1. Zero-shot: Models used as frozen feature extractors with a deterministic rule mapping hidden states to BIO tags (Begin, Inside, Outside).
  2. Fine-tuned: Models were fine-tuned with a token classification head on an 80/20 train/test split of ConLoan (10 epochs, batch size 16, AdamW optimizer).

C. Evaluation Metrics

Performance was measured using Precision, Recall, and F1-score via the seqeval library (evaluating at the entity span level).

• Strict Scoring: Predicted spans must match gold segmentation exactly.
• Relaxed Scoring: Gold multi-word spans are decomposed into tokens to accommodate LLM segmentation variability.

3. Key Results

LLM Performance (Prompting)

• Overall Failure: LLMs performed poorly across all settings, with average F1-scores below 0.50 (ranging from 0.23 to 0.47).
• Best Performer: Gemini achieved the highest average F1 (0.466), followed by OpenAI (0.41) and Llama (0.23).
• Prompt Sensitivity:
  • Adding few-shot examples had mixed results; it slightly improved recall for some prompts but degraded performance for others.
  • Prompt 3 (Usage-based) generally yielded lower scores, suggesting models struggle with the nuance of distinguishing loanwords from code-switching based on monolingual usage.
  • Anomaly: Llama under the few-shot Prompt 2 saw F1 scores drop near zero due to formatting failures (producing free-form text instead of lists), indicating fragility in instruction following.
• Language Variance: Romance languages (French, Italian, Spanish) and Chinese performed better than German, Icelandic, and Portuguese, likely due to morphological complexity and higher integration of loanwords in the latter.

Encoder Performance (Fine-tuning)

• Zero-shot Baseline: Unfine-tuned encoders performed extremely poorly (Average F1 < 0.02), confirming that pretraining alone is insufficient for this task.
• Fine-tuning Impact: Fine-tuning resulted in drastic improvements, with average F1-scores ranging from 0.648 to 0.851.
• Top Model: XLM-RoBERTa-Large achieved the best overall performance (F1 = 0.8513).
• Language Specifics: Fine-tuning was particularly effective for low-resource or morphologically complex languages (e.g., Northern Kurdish, Icelandic), boosting F1 from near-zero to >0.7.

4. Qualitative Analysis of Errors

The authors identified three systematic error patterns across both LLMs and fine-tuned models:

1. Code-Switching vs. Loanwords: Models frequently confused code-switching (e.g., using "really" in a Kurdish sentence) with integrated loanwords. While Prompt 3 helped, models still struggled to operationalize the sociolinguistic boundary between the two.
2. Named Entities: Proper nouns (e.g., "NASA," "PISA-Studie") were consistently misclassified as loanwords due to orthographic resemblance to foreign patterns, leading to high false-positive rates.
3. Scientific/Greco-Latin Vocabulary:
   • Fine-tuned models tended to under-detect scientific terms (e.g., "nitrates") as loanwords because they are morphologically regular and fully assimilated.
   • LLMs tended to over-detect them based purely on etymological cues (e.g., labeling "filosofia" as a loanword in Portuguese), failing to recognize their status as native lexical items in modern usage.

5. Key Contributions

1. Empirical Evidence of "Borrowing-Blindness": The study provides the first systematic evaluation showing that general-purpose LLMs fail to distinguish loanwords from native vocabulary, even with explicit instructions and definitions.
2. Multilingual Benchmarking: The work evaluates loanword identification across 10 diverse languages, highlighting that performance varies significantly based on language morphology and the degree of loanword integration.
3. Fine-tuning Necessity: The results demonstrate that while pretraining provides a foundation, task-specific fine-tuning is essential for loanword identification, with XLM-RoBERTa-Large achieving strong performance (F1 > 0.85) after training.
4. Error Taxonomy: The paper categorizes specific failure modes (confusion with code-switching, named entities, and technical terms), offering a roadmap for future model improvements.

6. Significance and Implications

• NLP for Minority Languages: The findings suggest that current NLP tools may inadvertently reinforce lexical pressure from dominant languages by failing to recognize or support native vocabulary. Accurate loanword identification is crucial for language preservation and educational tools.
• Model Limitations: The study reveals that LLMs lack a deep understanding of synchronic linguistic norms versus diachronic etymology. They rely heavily on surface-level orthographic cues rather than sociolinguistic integration or speaker intent.
• Future Directions: The authors suggest moving toward fine-grained identification tasks that categorize loanwords by their integration status (continuum approach) and developing models sensitive to pragmatic context and speaker intent.

In conclusion, the paper argues that loanword identification is far from a solved problem. While fine-tuned encoders show promise, current LLMs remain "borrowing-blind," unable to reliably differentiate between borrowed and native vocabulary without significant architectural or training adjustments.