Technical Summary: Culturally Entangled Homographs in Low-Resource LLMs

Problem Definition

The paper addresses a specific and underexplored variant of word-sense disambiguation (WSD) in the Bangla language: Culturally Entangled Homographs (CEH). Unlike standard WSD, where competing senses are distinct lexical definitions, CEH involves a single word form that functions simultaneously as a personal name and a culturally loaded common noun (e.g., Maya as a girl's name and as "affectionate compassion"; Arif as a boy's name and as "one who possesses intuitive knowledge of the divine" in Sufi theology).

Resolving CEH requires cultural and pragmatic knowledge rather than surface lexical statistics. The authors argue that current Large Language Models (LLMs), even those with language-specific pretraining, suffer from a systematic dominant-meaning bias. These models default to the common-noun sense and fail to recognize the personal-name reading, indicating a lack of genuine cultural grounding despite linguistic fluency.

Methodology

1. Dataset Construction (Bangla CEH Benchmark)

The authors constructed a benchmark of 1,516 expert-verified sentences containing 3,032 labelled occurrences.

• Structure: Each sentence contains a target word appearing twice with distinct readings (e.g., once as a Name, once as a Concept/Emotion/Spiritual State).
• Annotation: Every occurrence is assigned one of six culturally grounded categories (Name, Concept, Emotion, Emotional State, Collective State, Spiritual) and accompanied by a natural-language explanation of the cultural reasoning behind the label.
• Generation & Verification: Sentences were generated using three state-of-the-art models (Claude Fable 5, Kimi K3, Gemini 3.5 Flash) to ensure diversity. A two-stage verification protocol was applied: cross-model round-robin checking followed by manual review by two native Bangla speakers, who served as the final authority.

2. Experimental Regimes

The study evaluates models under four increasing levels of supervision:

1. Zero-Shot Prompting: Measures out-of-the-box cultural grounding without examples.
2. Few-Shot Prompting: Tests if in-context demonstrations help recognize entangled readings.
3. Cultural Chain-of-Thought (C-CoT): A contrastive prompting strategy where the model must explicitly argue for both candidate readings (Name vs. Concept) before committing to a decision. This aims to counteract the dominant-meaning bias without parameter updates.
4. Knowledge Distillation (KD) Fine-Tuning: Small models (1–3B parameters) are fine-tuned using QLoRA on the human-verified cultural explanations. Crucially, the training target is the full reasoning trace followed by the label, not the label alone, forcing the model to internalize the pattern of cultural reasoning.

3. Models Evaluated

The evaluation covers a mix of open-source and closed-source models:

• Open-Source: Qwen2.5-1.5B, Llama-3.2-3B, Gemma-3-1B, Ministral-3B, and TituLLM (a Bangla-specific model).
• Closed-Source: GPT-4o-mini.

4. Evaluation Metrics

• Label Track: Exact Match (EM), Average Score, Macro/Micro F1, Hallucination Rate, and Dominant-Bias Rate (frequency of missing the Name label).
• Reasoning Track: chrF and BERTScore against gold explanations, plus Human-as-a-Judge and LLM-as-a-Judge evaluations.

Key Results

1. Systematic Dominant-Meaning Bias

Under zero-shot prompting, all open-source models exhibited a severe bias toward the common-noun reading.

• Bias Rates: Ranged from 62.5% (Ministral) to 100% (Gemma and TituLLM).
• TituLLM Failure: The Bangla-specific model failed completely under every prompting regime (Zero-shot, Few-shot, C-CoT), hallucinating on 100% of inputs in some cases. This demonstrates that language-specific pretraining alone does not confer cultural grounding.
• GPT-4o-mini: While stronger (43.87% EM, 16.35% bias), it still struggled significantly compared to the task's requirements.

2. Limitations of Prompting

• Few-Shot: Improved performance for some models (e.g., Gemma's EM rose from 0.91% to 31.29%) but did not resolve the core bias for all models.
• C-CoT: Showed mixed results. It nearly eliminated bias for Qwen2.5-1.5B (dropping bias from 63.46% to 2.88%) but degraded performance for Llama-3.2-3B (bias increased to 75%) and TituLLM. This suggests C-CoT is highly model-dependent and requires the model to sustain long reasoning chains without format collapse.

3. Success of Knowledge Distillation

Fine-tuning small models on cultural reasoning traces yielded the most significant improvements:

• Performance Leap: All fine-tuned models exceeded 82% Exact Match and reduced dominant-meaning bias to under 5%.
• Reasoning Quality: BERTScore for reasoning improved drastically (e.g., Qwen from 0.178 to 0.784), indicating models learned to justify labels culturally rather than memorizing surface mappings.
• TituLLM Reversal: The previously failing Bangla-specific model became the strongest system (87.91% EM, 4.2% bias) after distillation, outperforming the fine-tuned general models and the prompting-only GPT-4o-mini.

Significance and Claims

The paper claims to provide a diagnostic tool for genuine cultural grounding in low-resource languages. Its primary contributions are:

1. Formalization of CEH: Isolating cultural grounding from surface lexical competence.
2. Benchmark Release: Providing a dataset with token-level cultural categories and reasoning explanations.
3. Empirical Evidence: Demonstrating that language-specific pretraining is insufficient for cultural tasks; a Bangla-specific model can fail completely without specific cultural supervision.
4. Practical Solution: Showing that distilling cultural reasoning into small, parameter-efficient models is a viable path to high performance. This approach allows small, open models to outperform larger proprietary models on culturally complex tasks once they are taught the reasoning process.

The authors conclude that for low-resource, culturally rich languages, the most effective strategy is pairing a compact language-specific model with a small set of cultural reasoning traces and adapting it via parameter-efficient fine-tuning. They acknowledge limitations, including the manual nature of word selection (reflecting a single annotator's knowledge) and the persistent difficulty models face with rarer cultural labels (long-tail data scarcity).