Technical Summary: IndoBias

Problem Statement

Large Language Models (LLMs) absorb and reproduce societal stereotypes from their pre-training corpora, often harming underrepresented groups. While significant effort has been invested in bias benchmarks for English (e.g., CrowS-Pairs, StereoSet), these resources fail to capture the unique sociocultural and linguistic complexities of non-English contexts. Specifically, Indonesia presents a critical gap: a nation with over 1,300 ethnic groups and 700 indigenous languages, where bias is rarely uniform. Existing evaluation frameworks often rely on widely recognized tropes, which structurally excludes smaller or less visible communities (e.g., the Korowai people) from bias assessment, perpetuating an "invisibility bias." Furthermore, it remains unclear how pre-training data sources (e.g., unfiltered web crawls vs. human-reviewed articles) and the inclusion of local languages influence the emergence of cultural bias in LLMs.

Methodology

To address these gaps, the authors introduce IndoBias, the first culturally grounded, dual-track benchmark for evaluating bias in Indonesian and three local languages: Javanese, Sundanese, and Makasar. The benchmark consists of two complementary evaluation tracks:

1. IndoBias-Pairs (Depth-Oriented)

This track utilizes a contrastive approach to measure the intensity of stereotypes.

• Construction: 544 manually curated sentence pairs were created in Indonesian, covering five bias domains: Identity and Demographics, Economic Status, Cultural and Geographic, Social and Family Roles, and Ideology and Religion. These were further divided into 18 subdomains.
• Multilingual Expansion: The pairs were translated into Javanese, Sundanese, and Makasar by native speakers, resulting in 4,352 total instances (2,176 prototypical and 2,176 counter-stereotypical statements).
• Evaluation Metric: The "Protrope Win Rate" measures the proportion of times a model assigns a lower perplexity (higher likelihood) to the stereotypical statement compared to its counter-stereotypical counterpart. A score of 50% indicates no bias; higher scores indicate stronger stereotypical bias.

2. IndoBias-QA (Breadth-Oriented)

This track uses a generation-based approach to assess representational fairness across a vast array of entities, including those lacking documented stereotypes.

• Framework: Built upon the LLM Stereotype Index (LSI) framework, combining four pivots: Demography, Demographic Group, Stereotype Pair, and Task ID.
• Scope: Evaluates 336 Indonesian demographic groups across six categories: Ethnicity (81), Government Institutions (60), Names (78), Political Parties (20), Religions (29), and Universities (68).
• Task Complexity: To circumvent safety filters and force substantive responses, the benchmark employs seven task formats of increasing complexity, ranging from simple forced-choice questions to code variable assignments and story writing.
• Stereotype Axes: The evaluation integrates social science frameworks: the Social Progress Index (SPI), O*NET (for skill-based traits), and Worldwide Governance Indicators (WGI) to define positive and negative stereotype pairs.
• Evaluation Metric: "Stereotype Polarity" (SP) measures the percentage of valid outputs mapped to a positive stereotype label.

3. Pretraining Simulation

To understand the emergence of bias during training, the authors conducted a controlled pretraining simulation using IndoBERT initialized from random weights.

• Setup: Models were trained for 500,000 steps across six different data compositions involving Indonesian Common Crawl (CC-100), Wikipedia, and Liputan 6 (news), with varying mixes of Javanese and Sundanese data.
• Analysis: Checkpoints were evaluated every 25,000 steps using IndoBias-Pairs to track bias evolution.

Key Results

Model Performance on IndoBias-Pairs

• Decoder vs. Encoder: Decoder models (generative LLMs) exhibited significantly stronger prototypical bias than encoder-only models. Many decoder models achieved protrope win rates exceeding 70% in Indonesian, whereas encoders rarely exceeded 67% and often hovered near 50%.
• Language Effects: Prompting in Indonesian generally yielded high bias scores. However, in the "Ideology and Religion" domain, local languages (particularly Makasar) sometimes exhibited higher bias than Indonesian.
• Fine-Tuning Impact: Fine-tuning models on local language data (e.g., Komodo-7B, SeaLLM-v3, Sailor2-8B) consistently increased prototypical bias. The authors hypothesize that improving fluency in local languages amplifies associations found in real-world training corpora.

Performance on IndoBias-QA

• Model Discrepancy: Newer, more capable models (e.g., Qwen3.5, GPT-5 mini) showed higher standard deviations in stereotype polarity (SPσ) across demographic groups compared to older models. This suggests that while newer models may not be uniformly more biased, their bias is more variable and context-dependent.
• Non-Uniform Bias: Bias varied significantly across demographic groups. For instance, models rated marginalized ethnic groups (e.g., Korowai) as materially worse off than major groups (e.g., Javanese). Similarly, government institutions and political candidates received disparate scores on metrics like "Control of Corruption" and "Opportunity," posing risks for fairness in downstream applications like hiring or electoral analysis.

Pretraining Findings

• Data Source: Unfiltered web corpora (CC-100) introduced significantly more bias during pretraining compared to human-reviewed sources like Wikipedia and news articles (Liputan 6).
• Multilingual Mixing: Introducing local languages (Javanese, Sundanese) into the pretraining mix generally increased bias across evaluated languages, with the exception of Indonesian evaluation in specific mixed-data scenarios.

Significance and Contributions

The paper claims three primary contributions:

1. IndoBias Benchmark: The first culturally grounded bias benchmark for the Indonesian context, covering four languages and utilizing both contrastive and generation-based evaluation tracks.
2. Comprehensive Evaluation: A broad benchmarking of encoder and decoder models (general, Southeast Asian, and Indonesian-specific) and an analysis of stereotype polarity across 336 demographic groups, revealing non-uniform bias patterns.
3. Pretraining Insights: A controlled simulation demonstrating that unfiltered web data introduces more bias than curated sources and that incorporating local languages into pretraining can inadvertently amplify cultural stereotypes.

The authors conclude that studying bias in culture-specific contexts is essential. They emphasize that standard evaluation frameworks often fail to capture the intricate web of linguistic and demographic prejudices in multilingual societies like Indonesia. IndoBias aims to support the development of fairer language technologies by highlighting these specific cultural and linguistic complexities.

Limitations

The authors acknowledge several limitations:

• Non-exhaustive Coverage: Despite covering 336 groups, this represents only a fraction of Indonesia's 1,300+ ethnic groups and 700+ languages.
• Structural Exclusion: The contrastive approach (IndoBias-Pairs) inherently excludes groups without widely recognized stereotypes, potentially perpetuating the invisibility of the most vulnerable communities.
• Language Scope: The study is limited to Indonesian, Javanese, Sundanese, and Makasar due to resource availability.
• Simulation Constraints: The pretraining simulation used a reduced batch size and a limited number of steps (500,000) compared to modern large-scale pretraining, which may affect the generalizability of the findings.