💬 NLP

Robust Bias Evaluation with FilBBQ: A Filipino Bias Benchmark for Question-Answering Language Models

This paper introduces FilBBQ, a culturally adapted Filipino bias benchmark comprising over 10,000 prompts that, when applied with a robust multi-seed evaluation protocol, reveals significant sexist and homophobic biases in Filipino language models regarding emotion, domesticity, queer interests, and polygamy.

Original authors: Lance Calvin Lim Gamboa, Yue Feng, Mark Lee

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lance Calvin Lim Gamboa, Yue Feng, Mark Lee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read robot that can speak many languages. You ask it questions, and it answers them like a human. But there's a catch: because this robot learned from the internet, it might have picked up some of society's unfair stereotypes, like "women are bad at math" or "gay men are obsessed with fashion."

This paper is about building a specialized test to see if this robot has these biases, but specifically for the Philippines and the Filipino language.

Here is the story of their work, broken down into simple concepts:

1. The Problem: The "One-Size-Fits-All" Test Didn't Fit

Scientists already had a famous test called BBQ (Bias Benchmark for Question-Answering). Think of BBQ like a standardized driving test. It's great for checking if a car (or a robot) knows the rules of the road in the US or Europe.

But the Philippines is a different country with different roads, different traffic signs, and different driving habits.

The Gap: The existing tests were mostly in English or other major languages. They didn't capture the unique stereotypes found in Filipino culture. For example, in the US, a stereotype might be about "denim overalls," but in the hot Philippines, that's not a thing. In the Philippines, a specific type of local helper (yaya) or a local term for a queer man (bakla) carries different cultural weight.
The Flaw in Testing: Previous tests asked the robot a question once and took the answer as the final truth. But robots are like nervous actors; if you ask them the same question twice in a row, they might give two different answers. Relying on just one answer is like judging a basketball player's skill based on a single shot.

2. The Solution: Building "FilBBQ" (The Filipino BBQ)

The researchers built a new test called FilBBQ. They didn't just translate the American test; they rebuilt it from the ground up to fit Filipino culture.

Think of this process like cooking a local dish:

Step 1: Sorting the Ingredients. They looked at the original American test and threw away ingredients that didn't make sense in the Philippines (like stereotypes about sports that aren't popular there).
Step 2: Localizing the Flavor. They translated the remaining questions but changed the names and situations. Instead of "Donna and Jermaine," they used popular Filipino names. Instead of "babysitter," they used "yaya" (nanny).
Step 3: Adding New Spices. They added brand new questions about stereotypes unique to the Philippines, like the idea that "tomboys" (masculine-presenting women) are good with cars, or that queer men are obsessed with fashion and gossip.
The Result: They created over 10,000 questions (prompts) to test the robot.

3. The New Rule: Don't Trust a Single Shot

The most important part of this paper is how they tested the robots.

Old Way: Ask the robot a question once. If it says "Women are emotional," you mark it as biased.
New Way (The Robust Protocol): The researchers asked the robot the same question 50 times (using different random seeds, like rolling dice to change the robot's mood).
Why? Because robots are unstable. Sometimes they might say "Women are emotional," and other times "Men are emotional," or "I don't know." By averaging the results of 50 tries, they got a true picture of the robot's personality, rather than just catching it on a bad day.

4. What They Found (The Taste Test Results)

When they ran this new, rigorous test on three different Filipino-speaking robots, they found some interesting things:

The "Emotional" Stereotype: The robots consistently thought that if a situation was vague, the person being emotional was likely a woman.
The "Domestic" Stereotype: The robots strongly associated women with being nurses, homemakers, and taking care of the family, while associating men with being doctors or breadwinners.
The "Queer" Stereotype: The robots often assumed that gay men were interested in fashion, design, and gossip, or that they struggled with monogamy (being faithful).
The "Big Data" Surprise: The robot that had read the most Filipino text (the biggest brain) actually showed the strongest biases. It seems that the more a robot reads about the world, the more it absorbs the world's unfair stereotypes, too.

5. Why This Matters

This paper is like a mirror held up to technology.

It shows us that if we want AI to be fair in the Philippines, we can't just use American tests. We need to understand local culture.
It teaches us that we can't trust a robot's answer if we only ask it once. We need to ask it many times to see the truth.
It gives us a tool (FilBBQ) to check if future robots are being fair to Filipino women and the LGBTQ+ community.

In a nutshell: The researchers built a culturally specific "lie detector" for robots in the Philippines and proved that to get an honest answer, you have to ask the same question over and over again. They found that even our smartest robots still carry the heavy baggage of old-fashioned stereotypes.

1. Problem Statement

The paper addresses two critical gaps in the current landscape of Natural Language Processing (NLP) bias evaluation:

Linguistic and Cultural Underrepresentation: Existing bias benchmarks, particularly the Bias Benchmark for Question-Answering (BBQ), are predominantly focused on high-resource languages from economically developed nations (e.g., English, Chinese, Japanese, German). There is a significant lack of benchmarks for low-resource languages from developing nations, specifically Filipino, despite the Philippines having high AI adoption rates. Furthermore, direct translations of Western benchmarks often fail to capture local sociocultural nuances, such as specific gender roles, local terminology for sexual orientation, and unique societal stereotypes.
Evaluation Instability: Previous studies using BBQ and its adaptations typically evaluate models using a single inference pass per prompt. However, generative Pre-trained Language Models (PLMs) are known to exhibit response instability, where the same prompt yields different outputs depending on the random seed. Relying on a single run leads to unreliable bias scores that may reflect outlier behaviors rather than the model's true aggregate tendencies.

2. Methodology

The authors propose FilBBQ, a culturally adapted bias benchmark, and a robust evaluation protocol to address the issues above.

A. Dataset Construction (FilBBQ)

FilBBQ was developed through a four-phase process, adapting the original English BBQ framework:

Template Categorization: Original BBQ templates were categorized into three groups: Remove (irrelevant to the Philippines, e.g., sports fairness regarding transgender individuals), Translate (culturally neutral), and Modify (culturally relevant but requiring local context adjustments).
Culturally Aware Translation:
- Demographic Labels: Replaced Western sexual orientation labels (e.g., gay, lesbian) with local Filipino terms (bakla, bading, tomboy, lesbiyana) which better reflect local identity constructions. Heterosexual labels were mapped to lalaki (male) and babae (female).
- Proper Names: Retained some American names due to colonial influence but integrated frequent Filipino names from the Philippine Statistics Authority. Surnames were replaced with common Filipino surnames.
- Contextual Adaptation: Replaced culturally specific references (e.g., "dark denim overalls" for lesbians, "babysitter") with local equivalents (e.g., "dark tee/pants/rubber shoes," "yaya/katulong").
New Template Construction: Created 52 entirely new templates based on academic literature and local media regarding Philippine-specific stereotypes (e.g., tomboys being good with cars, polygamy assumptions regarding queer individuals).
Prompt Generation: Automated scripts generated 10,576 prompts from 123 templates (77 gender, 46 sexual orientation). This includes both ambiguous (testing bias via the "unknown" option) and disambiguated (testing bias via specific social group choices) contexts.

B. Robust Evaluation Protocol

To mitigate response instability, the authors moved away from single-pass evaluation:

Multi-Seed Execution: Each model was queried 50 times for every prompt using different random seeds.
Aggregation: Bias scores ( $s_{dis}$ for disambiguated and $s_{amb}$ for ambiguous contexts) were calculated for each seed run and then averaged to produce a final, robust bias score.
Bias Profiling: The study formalized the reporting of the top 5 strongest biases per model, creating a detailed "bias profile" rather than a single aggregate metric.

C. Models Evaluated

Three models capable of processing Filipino were tested:

Llama-SEA-LION-v2-8B-IT: Continually pre-trained on Southeast Asian data (~1.24B Filipino tokens).
SeaLLMs-v3-7B-Chat: Fine-tuned for instruction following with safety enhancements.
roberta-tagalog-base: A masked language model trained on a purely Filipino dataset.

3. Key Contributions

FilBBQ Benchmark: The first culturally aware, QA-specific bias benchmark for the Filipino language, containing over 10,000 prompts and 52 locally specific templates.
Robust Evaluation Framework: A methodological shift demonstrating that multi-seed averaging is necessary to accurately capture model bias, proving that single-run evaluations are insufficient due to high variance in generative models.
Comprehensive Bias Profiles: The first detailed bias profiles for Filipino-capable PLMs, revealing specific patterns of sexism and homophobia rooted in Philippine culture.

4. Results

The evaluation yielded several significant findings:

High Variability: Bias scores varied significantly across the 50 seeds for the same prompts. For example, a prompt regarding "women are emotional" yielded scores ranging from -1.00 (counter-bias) to 1.00 (extreme bias) across seeds, confirming the necessity of the multi-run protocol.
Dominant Biases:
- Sexism: All models exhibited strong biases linking women to emotionality and domestic roles (e.g., nurses, homemakers) while associating men with leadership and economic provision. Llama-SEA-LION showed the strongest bias in this category (0.78 score for "Women are nurses").
- Homophobia: Models displayed biases associating queer individuals with stereotypical interests (fashion, gossip, design) and polygamy/non-monogamy.
- Specific Model Anomalies: roberta-tagalog-base (a masked model) showed an alarming bias score (0.16) linking non-heterosexual individuals to pedophilia, a stereotype not as prominent in the other models.
Correlation with Pretraining: The model with the largest Filipino pretraining corpus (Llama-SEA-LION) exhibited the highest bias scores across the most prompts, suggesting a potential positive correlation between the volume of language-specific training data and the entrenchment of societal biases within the model.

5. Significance

Cultural Relevance: FilBBQ proves that bias is not universal; it is deeply contextual. A benchmark must be culturally adapted to detect biases like the specific local conceptions of bakla or tomboy identities, which Western benchmarks miss.
Methodological Rigor: The paper establishes that response instability is a critical factor in bias evaluation. Future studies must adopt multi-seed averaging to avoid drawing conclusions from statistical noise or outliers.
Resource for the Global South: By focusing on a low-resource language from a developing nation, the paper challenges the dominance of high-resource language bias research and provides a template for creating similar benchmarks for other underrepresented languages.
Ethical Implications: The findings highlight that simply increasing training data volume does not reduce bias; in fact, it may amplify societal prejudices present in the training corpus, necessitating targeted debiasing interventions.