Language Ideologies in a Multilingual Society: An LLM-based Analysis of Luxembourgish News Comments

1. Problem Statement

The paper addresses the challenge of automatically detecting language ideologies—shared beliefs about language that shape social identities, power dynamics, and belonging—within large-scale text corpora. Specifically, the authors focus on Luxembourgish, a low-resource West Germanic language spoken in a highly multilingual society.

• The Core Challenge: Language ideologies are complex, context-dependent, and often subtle. Traditional manual annotation is time-consuming and difficult to scale.
• The Low-Resource Barrier: Luxembourgish is underrepresented in Large Language Model (LLM) training data. It is unclear whether LLMs can effectively perform fine-grained sociolinguistic tasks in such languages or if machine translation to high-resource languages (English, French, German) is necessary to improve performance.
• Research Questions:
  1. Can current LLMs perform multi-class language ideology classification?
  2. Does translating low-resource data to high-resource languages improve classification accuracy?
  3. Can LLMs contribute to sociolinguistic research without heavy technical adaptation?

2. Methodology

Data Collection and Annotation

• Corpus: 300 user comments from the Luxembourgish national broadcaster (RTL.lu), split into two subsets:
  • Lang (150 comments): Selected via keyword matching related to language use (e.g., "our language," "mother tongue").
  • NotLang (150 comments): Selected to be unrelated to language use to prevent models from relying on surface-level lexical cues.
• Preprocessing: Comments were split into sentences using GPT (to handle irregular social media punctuation/emojis) and manually corrected. No lexical normalization was applied to preserve orthographic variations that might signal ideology.
• Gold Standard Annotation: Two expert linguists annotated 1,524 sentences at the sentence level using six categories:
  1. Identity: Statements on national/personal identity linked to language.
  2. Vitality: Statements on language endangerment, decay, or preservation.
  3. Belonging: Statements on requirements for integration into society.
  4. Responsibility: Statements assigning blame or duty for the linguistic situation.
  5. Recognition: Statements on language status, policy, and hierarchy.
  6. None: No ideology present.
• Inter-annotator Agreement: Cohen's Kappa was 0.66 (0.78 raw agreement).

Experimental Setup

• Models Evaluated: A mix of frontier and open-weight models, including GPT-4o, GPT-4o-mini, o3, GPT-5, DeepSeek-V3, Qwen3, Mistral Magistral, Aya Expanse, and Llama 4.
• Prompt Engineering: Four prompt configurations were tested, ranging from simple label lists to complex prompts containing:
  • Detailed category definitions.
  • Positive and negative examples (few-shot).
  • Instructions to output JSON with justifications.
  • Result: Prompt 4 (definitions + examples + instructions) yielded the best performance.
• Cross-Lingual Transfer (Translate-Test): To address the low-resource nature of Luxembourgish, the dataset was machine-translated into English, French, and German using Google Translate.
  • Quality Control: Two translation sets were created: Automatic (raw) and Post-Edited (manually corrected for country/language mentions and orthography).
• Evaluation Metrics: Weighted F1, Macro F1, and per-label F1 scores.

3. Key Results

Prompt Engineering Impact

• Providing detailed definitions and examples (Prompt 4) significantly improved performance over simple prompts.
• F1 scores improved from ~0.43 (Prompt 1) to ~0.56 (Prompt 4) for GPT-4o.
• GPT-5 consistently outperformed all other models across all languages and settings.

Cross-Lingual Transfer Findings

• No Significant Gain from Translation: Contrary to the hypothesis that translating to high-resource languages would boost performance, translations did not lead to substantial improvements.
  • In some cases (e.g., DeepSeek), the original Luxembourgish text performed better than translations.
  • There was no significant difference between Automatic and Post-Edited translations.
• Reasoning: The authors suggest that while models generalize from shared multilingual representations, the sociocultural nuances and specific ideological markers of Luxembourgish are often lost or distorted during translation, negating the benefits of the model's better "understanding" of the target language.

Fine-Grained Classification Performance

• Binary vs. Multi-Class: LLMs performed well at distinguishing ideological content from non-ideological content (the "None" label achieved F1 scores ~0.9).
• Category Confusion: Fine-grained classification between the five specific ideologies was challenging.
  • Over-prediction: Models frequently confused Vitality, Identity, and Belonging.
  • Under-detection: Responsibility and Recognition were often under-detected.
  • Specific Confusions:
    • Identity vs. Belonging: Models struggled to distinguish between national self-determination (Identity) and social exclusion/integration requirements (Belonging).
    • Vitality vs. Recognition: Calls to preserve the language were often mislabeled as status recognition rather than vitality concerns.
• Model Explanations: The models' generated rationales revealed that they often relied on superficial cues (e.g., first-person pronouns like "we/mir") rather than deep semantic understanding of the ideological stance.

4. Key Contributions

1. Evaluation of LLMs on Low-Resource Ideology Detection: The study provides empirical evidence that while LLMs are effective at detecting the presence of language ideology in low-resource languages, they struggle with fine-grained categorization without human expertise.
2. Translation Paradigm Analysis: It challenges the standard "translate-test" assumption in low-resource NLP for sociolinguistic tasks, demonstrating that translation can degrade performance by stripping away culturally specific ideological markers.
3. Prompt Engineering for Sociolinguistics: The paper demonstrates that providing detailed definitions and examples (few-shot learning) is crucial, but even with these, LLMs require expert guidance to resolve category overlaps.
4. Dataset and Annotation Schema: The creation of a manually annotated corpus of Luxembourgish comments with a specific five-category ideological framework serves as a benchmark for future computational sociolinguistics research.

5. Significance and Limitations

• Significance: The research validates the use of LLMs as a pre-filtering tool for sociolinguists. They can efficiently identify texts containing language ideologies (binary classification) to reduce the manual workload, but they cannot yet replace human experts for nuanced ideological analysis.
• Limitations:
  • Data Scarcity: The small dataset (300 comments) prevented fine-tuning models, relying solely on in-context learning.
  • Category Overlap: The ideological categories are inherently fuzzy and overlap in human discourse, making them difficult for models to disentangle without more sophisticated training.
  • Prompt Sensitivity: The study noted that minor grammatical errors in the prompt (e.g., incorrect pronoun forms) could lead to systematic labeling errors.
• Future Work: The authors suggest moving toward mixed-methods approaches, where LLMs handle the initial bulk analysis and human experts refine the results. They also propose simplifying the annotation schema or collecting more data for fine-tuning to improve performance.

In conclusion, the paper argues that while LLMs are powerful tools for scaling sociolinguistic research, they currently lack the deep cultural and contextual understanding required to fully automate the detection of complex language ideologies in low-resource, multilingual settings.