Cross-Lingual Probing and Community-Grounded Analysis of Gender Bias in Low-Resource Bengali

Technical Summary: Cross-Lingual Probing and Community-Grounded Analysis of Gender Bias in Low-Resource Bengali

Problem Statement
While Large Language Models (LLMs) have advanced Natural Language Processing (NLP), they perpetuate intrinsic gender biases that exacerbate social inequalities. Existing research and bias mitigation frameworks are heavily skewed toward English and other high-resource Indo-European languages. Low-resource languages, particularly Bengali, remain under-examined. Bengali presents unique challenges: it lacks grammatical gender categories yet encodes implicit and explicit biases through cultural norms, syntax, and morphology. Consequently, direct application of English-centric bias detection tools to Bengali is constrained by linguistic disparities and socio-cultural factors, creating a critical blind spot for NLP systems serving the Global South.

Methodology
The study employs a multi-faceted approach to identify and analyze gender bias in Bengali, combining computational extraction techniques with community-grounded field investigations.

1. Data Sourcing and Extraction:
   The authors utilized four complementary data sources to create a pipeline for bias identification:

   • CORGI-PM: A Chinese corpus of 32.9K sentences (5.2K labeled biased) used for translation-based transfer and binary classification supervision.
   • BUG: A large-scale English resource (108K sentences) used to test cross-lingual transfer of stereotypes via translation.
   • YouTube Comments: 7,340 organically produced Bengali comments collected via targeted queries on gender controversies, serving as a real-world evaluation bed.
   • FSB (GPT-Generated): 1,000 Bengali texts generated by GPT-3.5-Turbo using systematic prompting, annotated for bias to train continuous bias scorers.

2. Computational Approaches:
   Six distinct methods were implemented to extract biased utterances:

   • LaBSE Mining: Cross-lingual retrieval using LaBSE embeddings to find Bengali sentences similar to high-bias English seeds.
   • Translation-Based Transfer: Translating biased sentences from CORGI-PM and BUG into Bengali to assess bias retention.
   • CORGI Classifier: Fine-tuning mBERT on the CORGI-PM dataset to classify Bengali YouTube comments.
   • FSB Scorer: Optimizing BanglaBERT with LoRA on GPT-generated data to predict continuous bias scores.
   • CORGI-style Mining: Lexicon-based association scoring using mBERT and BanglaBERT embeddings to link adjectives to gendered concepts.
   • GPT Generation: Generating English sentences via prompts and translating/adapting them to Bengali.

3. Community-Grounded Analysis:
   To address the limitations of automated systems, the authors conducted field investigations in rural and low-income areas of Bangladesh. These studies involved interactive activities (e.g., role-play, skits) to elicit authentic insights on gender bias from participants who might not align with digital or formal linguistic definitions of bias.

Key Results
The study evaluated the "yield" (proportion of validated biased text) across different methods:

• GPT-Based Generation: Achieved the highest yield (90%) of potentially biased text but suffered from low topical and idiomatic diversity, often appearing repetitive and culturally shallow.
• Translation-Based Transfer: Retained bias in only 24% (from BUG) to 33% (from CORGI-PM) of translated sentences, indicating that many bias-bearing terms do not transfer naturally due to lexical and cultural mismatches (e.g., formal vs. colloquial terms for "teacher" or "housewife").
• Cross-Lingual Retrieval (LaBSE): Yielded approximately 18–20% biased candidates, limited by the Anglocentric nature of the source seeds and structural differences between English and Bengali corpora.
• Classification Models: The CORGI classifier flagged 4.67% of YouTube comments as biased, but manual verification confirmed only 36% of these as true positives, highlighting significant domain-shift issues when moving from curated corpora to colloquial social media.
• Lexicon-Based Mining: Produced extremely low yields (3% for mBERT, 0% for BanglaBERT), demonstrating the brittleness of purely heuristic approaches in capturing nuanced Bengali stereotypes.

Significance and Claims
The paper claims to be one of the first systematic studies of gender bias specifically within the Bengali language context. Its primary contributions are:

• Demonstration of Limitations: It empirically shows that English-centric bias detection frameworks (like CrowS-Pairs or StereoSet) and direct translation strategies fail to capture the full scope of gender bias in Bengali due to linguistic structures and cultural contexts.
• Necessity of Localization: The findings argue that bias in Bengali presents distinct characteristics requiring localized, context-sensitive methodologies rather than direct adaptation of Global North tools.
• Community Integration: The study emphasizes that automated systems often miss culturally relevant biases. It advocates for integrating community-driven research approaches, particularly from rural and low-income demographics, to validate and expand bias datasets.
• Foundation for Future Work: By highlighting the need for linguistically specific tools and intersectional perspectives (considering caste, religion, and geography), the paper establishes a foundation for developing more inclusive and fair NLP systems for Bengali and other Indic languages.

The authors conclude that while computational methods can generate or retrieve biased data, they currently lack the cultural grounding to be fully effective. Future progress depends on participatory data collection, improved translation systems, and the development of lexicons and prompts that are deeply rooted in the specific socio-cultural realities of Bengali speakers.