JUBAKU: An Adversarial Benchmark for Exposing Culturally Grounded Stereotypes in Japanese LLMs

1. Problem Statement

Large Language Models (LLMs) encode social biases that are deeply rooted in cultural norms. While numerous benchmarks exist for evaluating social bias in English (e.g., CrowS-Pairs, BBQ, StereoSet), existing evaluations for non-English contexts, particularly Japanese, suffer from two critical limitations:

• Cultural Misalignment: Current Japanese benchmarks are often direct translations of Western datasets. They fail to capture Japan-specific stereotypes related to hierarchical relationships, regional dialects, indirect communication styles, and traditional gender roles.
• Lack of Adversarial Rigor: Standard evaluations often rely on static datasets that do not actively probe for latent biases. Models may appear unbiased in standard settings but fail when presented with adversarial inputs designed to trigger implicit cultural assumptions.

The authors argue that relying on translated Western benchmarks leads to a false sense of safety, as these models may still harbor harmful, culturally specific biases that remain undetected.

2. Methodology: The JUBAKU Benchmark

To address these gaps, the authors introduce JUBAKU (Japanese cUlture adversarial BiAs benchmarK Under handcrafted creation), a benchmark specifically designed to expose latent biases in Japanese LLMs through adversarial dialogue scenarios.

A. Cultural Framework

The benchmark is structured around ten distinct cultural categories derived from a comprehensive survey on measuring culture in LLMs (Adilazuarda et al., 2024). These categories are:

1. Gender
2. Religion
3. Ethnicity
4. Education
5. Race
6. Region
7. Emotions and Values
8. Food and Drink
9. Basic Actions
10. Names

B. Adversarial Data Construction Process

Unlike benchmarks generated by LLMs (which risk circularity) or simple translations, JUBAKU instances are hand-crafted by native Japanese annotators and refined using an adversarial loop:

1. Drafting: Six native Japanese annotators (NLP researchers) created dialogue contexts and two candidate responses (one biased, one unbiased) for each cultural category. The biased response relies on unstated cultural stereotypes (e.g., assuming a white exchange student speaks English).
2. Adversarial Refinement: The instances were tested against GPT-4o. The goal was to iteratively edit the context until GPT-4o incorrectly selected the biased response as the appropriate one.
3. Finalization: Only instances that successfully "tricked" GPT-4o into choosing the biased option were retained. This ensures the benchmark is robust enough to challenge even state-of-the-art models.
4. Augmentation: To ensure generalizability and mitigate positional bias, the 152 base instances were augmented with different task instructions, few-shot examples, and answer order swaps, resulting in a final test set of 1,216 instances.

C. Evaluation Protocol

• Task Format: Binary choice. Given a dialogue context and two responses (A and B), the model must select the unbiased response.
• Metric: Accuracy (proportion of times the model selects the unbiased response). In this adversarial setting, low accuracy indicates high bias.
• Models Evaluated: Nine Japanese LLMs (Sarashina2, Qwen2.5, Swallow, calm3) across various parameter sizes (7B to 72B).
• Comparators: The models were also tested on three existing Japanese benchmarks: JBNLI, JBBQ, and SSQA-JA.

3. Key Results

The evaluation yielded stark contrasts between JUBAKU and existing benchmarks:

• Performance Collapse on JUBAKU: All nine Japanese LLMs performed below the random baseline (50%) on JUBAKU, with average accuracies ranging from 13% to 33%. This indicates that the models consistently preferred biased responses over unbiased ones when challenged with culturally grounded adversarial prompts.
• Superficial Performance on Existing Benchmarks: The same models achieved significantly higher accuracy (often >50% to >80%) on the existing benchmarks (JBNLI, JBBQ, SSQA-JA). This suggests that current benchmarks fail to detect the latent biases that JUBAKU exposes.
• Human Validation: Native human annotators achieved 91% accuracy in identifying the unbiased responses, confirming that the dataset contains clear, unambiguous correct answers and that the models' failures are due to bias, not ambiguity.
• Category Sensitivity: Performance varied by category. Models struggled most with Religion (21.3% accuracy) and Ethnicity (22.8% accuracy), requiring more adversarial edits to trick GPT-4o in these domains, suggesting these are highly sensitive areas where safety alignment is fragile.
• Generalizability: Adversarial edits designed to fool GPT-4o also caused significant accuracy drops in other models, proving that the adversarial construction method is effective across different architectures.

4. Key Contributions

1. First Culturally Grounded Adversarial Benchmark for Japanese: JUBAKU is the first benchmark to move beyond translation and utilize native cultural norms to construct adversarial examples.
2. Adversarial Construction Methodology: The paper demonstrates a novel pipeline where human annotators draft data, and an LLM (GPT-4o) is used iteratively to refine the data until it successfully triggers a bias, ensuring high difficulty and relevance.
3. Empirical Evidence of Latent Bias: The study provides concrete evidence that Japanese LLMs, despite appearing safe on standard benchmarks, harbor deep-seated cultural stereotypes that are only revealed through adversarial, culturally specific testing.
4. Open Resource: The dataset and code are released to the community to facilitate further research into culturally aligned AI safety.

5. Significance and Implications

• Safety vs. Utility: The paper highlights that "safety" in LLMs is not universal; a model can be safe in Western contexts but harmful in Japanese contexts due to unaddressed cultural norms (e.g., indirectness, hierarchy).
• Limitations of Translation: The results strongly argue against the practice of simply translating Western bias benchmarks for other languages. Local cultural nuances require localized, hand-crafted evaluation strategies.
• Future Directions: The authors note that JUBAKU currently focuses on "safety" (avoiding biased outputs) rather than "utility" (helpfulness). Future work must balance these two aspects, ensuring models are not only unbiased but also culturally appropriate and helpful.
• Ethical Considerations: The dataset contains sensitive stereotypes but is strictly intended for academic research to improve model safety. The authors emphasize that the stereotypes are used solely to expose and mitigate bias, not to promote them.

In conclusion, JUBAKU serves as a critical diagnostic tool, revealing that current Japanese LLMs are significantly more biased than previously thought when evaluated against culturally grounded adversarial scenarios. It calls for a paradigm shift in how non-English AI safety is evaluated, moving from translation-based metrics to culturally native, adversarial frameworks.