Bridging the Culture Gap: A Framework for LLM-Driven Socio-Cultural Localization of Math Word Problems in Low-Resource Languages

1. Problem Statement

Large Language Models (LLMs) have demonstrated strong capabilities in solving Mathematical Word Problems (MWPs), but their performance in low-resource languages (LRLs) is often evaluated using benchmarks that are merely direct translations of English datasets (e.g., GSM8K). These translated benchmarks suffer from English-centric bias:

• Cultural Mismatch: They retain English-specific entities (names like "Mandy," currencies like "$," and organizations) that do not reflect the socio-cultural reality of the target language speakers.
• Evaluation Flaws: Current benchmarks fail to test if models can reason correctly when the problem context is grounded in local culture.
• Scalability Issues: Creating truly localized datasets via human annotation is prohibitively expensive and slow, leading to a scarcity of high-quality, culturally adapted training and evaluation data for LRLs.

The core research question is: Do LLMs possess true multilingual mathematical reasoning, or do they rely on English-centric patterns that break when native cultural entities are introduced?

2. Methodology

The authors propose an LLM-driven automated localization framework to bridge this gap. The pipeline transforms existing translated benchmarks into culturally grounded variants through a multi-stage process:

A. The Localization Pipeline

The framework operates on three versions of an input sample: the original English ($x_{en}$), the direct translation ($x_{trans}$), and an entity-replaced version ($x_{ent}$).

1. Entity Classification: Using an LLM (specifically Gemini models with structured output), the system identifies key entities in the text: personal names, organization names, and currencies. The authors found LLMs superior to traditional NER/POS tagging for handling spelling variations and casing in LRLs.
2. Multilingual Entity Database: A curated database of unisex local names, organizations, and currency units was created for 18 African languages (to avoid gender bias).
3. Replacement Dictionary: The system maps English entities to culturally appropriate local equivalents (e.g., "$"$\to$ "shilingi," "Mandy" $\to$ "Camari").
4. Auto-Localization (The Core Step): Instead of asking the LLM to translate from scratch, the system uses a one-shot prompting strategy. It provides the LLM with the pair $(x_{en}, x_{trans})$ as a context example and asks it to translate the entity-replaced English sentence ($x_{ent}$) into the target language. This ensures the structural integrity of the problem is preserved while swapping cultural entities.
5. Quality Control:
   • Length & Similarity Checks: Ensures the localized output matches the length of the direct translation and has high semantic similarity (using difflib).
   • Entity Verification: Confirms that replacement entities are present and no unintended entities were added.
   • Fallback Mechanism: If quality checks fail, the system reverts to the original direct translation to prevent noise.

B. Datasets

• Source: GSM8K (English) and AfriMGSM (manually translated to 18 African languages).
• Training Data: Generated by applying the pipeline to GSM8K, filtered by translation quality (SSA-COMET score > 0.65), resulting in ~25k localized examples.
• Evaluation Data: A localized version of the AfriMGSM test set where entities were swapped for native equivalents.

C. Experimental Setup

• Models Evaluated: A mix of open-source (Gemma-2-9b-it, LLaMA-3-8B/70B, Mistral, Aya) and closed-source (GPT-4o-mini) models.
• Metrics: Numeric Match (NM), which checks if the predicted number matches the ground truth, ignoring formatting.
• Research Questions (RQs):
  • RQ1: Can LLM pipelines effectively localize benchmarks?
  • RQ2: Does introducing local entities reveal performance disparities?
  • RQ3: Can fine-tuning on localized data improve model robustness?

3. Key Contributions

1. Automated Localization Framework: A scalable pipeline that generates culturally adapted MWPs for 18 low-resource languages without relying on expensive human annotators for every step.
2. Discovery of Cultural Bias: The study empirically demonstrates that standard translated benchmarks mask true model capabilities. Models often perform significantly worse when native entities are introduced, revealing an over-reliance on English-centric patterns.
3. Performance Gaps: The authors quantified a performance drop of up to 9% (Numeric Match) for models like GPT-4o-mini when evaluated on localized vs. translated data.
4. Robustness via Fine-Tuning: The paper demonstrates that fine-tuning models on the generated localized datasets significantly improves their robustness and generalization across languages, effectively "teaching" the model to handle cultural variations.

4. Results and Analysis

RQ1: Effectiveness of the Pipeline

• Human Validation: The automated pipeline achieved high inter-annotator agreement (Cohen's kappa ~0.79–0.84) compared to direct prompting.
• Quality: The pipeline produced significantly more culturally appropriate outputs than direct prompting, which often failed to replace entities or reverted to the original translation.

RQ2: Performance Disparities (The "Culture Gap")

• Translation vs. Localization: Models generally performed better on direct translations (which contain English entities) than on auto-localized versions.
• English Benchmark Sensitivity: Even when tested on English prompts, replacing English entities with native ones (e.g., changing "Mandy" to a local name) caused performance drops in several models (e.g., Aya-expanse-32b), proving that models are sensitive to entity grounding, not just language.
• Language Variance: The impact varied by language and model. For instance, French and Swahili showed pronounced effects in Deepseek and Mistral models.

RQ3: Improving Robustness

• Fine-Tuning Impact: Fine-tuning on the localized datasets ($\hat{x}_{loc}$) improved model robustness.
• Hybrid Approach: The best results were achieved by combining translated data, entity-replaced data, and fully localized data. This suggests that diverse training sources help models generalize better to cultural variations.
• Model Specifics: Gemma-2-9b-it showed consistent gains from localized data, while LLaMA-3-8B-Instruct showed mixed results, highlighting that model architecture influences cultural adaptability.

5. Significance and Future Work

• Re-evaluating Benchmarks: The paper argues that current multilingual benchmarks are flawed because they do not account for socio-cultural context. A model passing a translated benchmark may fail in real-world applications where local entities are used.
• Scalable Solution: The framework provides a cost-effective method to generate high-quality, culturally grounded datasets for LRLs, addressing the data scarcity bottleneck.
• Future Directions: The authors suggest extending this approach beyond mathematics to other reasoning tasks and focusing on creating problems rooted in specific community activities rather than just swapping names.

Conclusion:
This work establishes that mathematical reasoning in LLMs is not culture-free. By introducing an automated framework for socio-cultural localization, the authors demonstrate that current models are brittle when faced with native entities. However, by augmenting training data with these culturally adapted variants, model robustness can be significantly improved, paving the way for more reliable and equitable AI deployment in low-resource linguistic contexts.