A Japanese Benchmark for Evaluating Social Bias in Reasoning Based on Attribution Theory

1. Problem Statement

The paper addresses two critical limitations in current social bias evaluation for Large Language Models (LLMs), specifically within the Japanese cultural context:

• Cultural Misalignment: Existing Japanese benchmarks (e.g., JBNLI, JBBQ) largely rely on translating English datasets. This approach fails to capture stereotypes and social norms unique to Japanese culture, as bias is deeply rooted in specific cultural contexts.
• Focus on Conclusions vs. Reasoning: Current benchmarks primarily evaluate the conclusion of an LLM's output. However, recent studies suggest that even when aligned models produce superficially neutral conclusions, social bias often persists within the reasoning process. Existing methods fail to isolate and detect these latent biases in the reasoning chain.

2. Methodology

The authors propose a new framework and dataset, JUBAKU-v2, designed to evaluate bias specifically in the reasoning phase while controlling for the final output.

A. Theoretical Foundation: Attribution Theory

The study is grounded in Attribution Theory from social psychology, specifically Pettigrew's Ultimate Attribution Error.

• Definition of Bias: A response is considered biased if it:
  1. Attributes an individual's undesirable behavior or characteristics solely to internal factors (personality, ability, intent) based on group attributes (e.g., nationality, gender) without factual verification.
  2. Attributes an individual's achievements to external factors while underestimating their internal abilities.
• Neutral Reasoning: Relies on situational factors (e.g., proximity, immediate context) rather than group stereotypes.

B. Dataset Construction (JUBAKU-v2)

The dataset was constructed to eliminate confounding factors (such as the utility of a proposal) that existed in previous versions.

1. Seed Data: Used the JUBAKU-v1 dataset as a base.
2. LLM Generation & Human Revision:
   • Standardization: Used GPT-5.1 to rewrite dialogues so that both the "Biased" and "Neutral" options lead to the exact same conclusion/proposal. The only difference lies in the reasoning (attribution).
   • Selection: Generated cases were tested on GPT-4o; cases where the model selected the biased option were flagged as "bias-inducing."
   • Human Validation: Six native Japanese speakers reviewed the cases. Only cases with high inter-annotator agreement (3/6 or more selecting the neutral option) were retained.
3. Final Composition:
   • Total Examples: 216 evaluation cases (derived from 27 unique scenarios).
   • Categories: 10 cultural categories including Gender, Religion, Nationality/Ethnicity, Education, Race, Region, Emotions/Values, Food/Drink, Basic Behaviors, and Name.
   • Structure: Each scenario presents a context and two reasoning paths (Biased vs. Neutral) leading to the same conclusion. The model must select the unbiased reasoning.

C. Experimental Setup

• Models Evaluated: 9 models, including top-tier API models (GPT-4o, GPT-5.2, Claude 4 Sonnet) and open-weight models (Qwen3 series, GPT-OSS series).
• Benchmarks: Compared against JBNLI, JBBQ, and SSQA-JA.
• Metrics:
  • Accuracy: Proportion of correctly selected neutral responses.
  • Discriminability: Variance in scores across models to see if the benchmark can distinguish between high-performing models.
  • Robustness: "Answer Fluctuation Rate" measuring consistency across 8 different prompt variations (instruction templates + order permutations).

3. Key Contributions

1. JUBAKU-v2 Dataset: The first Japanese benchmark specifically designed to evaluate social bias in reasoning based on attribution theory, contextualized for Japanese culture.
2. Differentiation of High-Performance Models: Demonstrated the ability to detect performance gaps between top-tier models (e.g., GPT-4o vs. GPT-5.2) that often saturate existing benchmarks.
3. Robustness Evaluation: Introduced a methodology to evaluate model consistency against prompt variations, revealing significant instability in some models.

4. Key Results

• Performance Gaps:
  • GPT-4o scored 81.9% on JUBAKU-v2, significantly lower than Claude 4 Sonnet (99.1%) and GPT-5.2 (98.1%).
  • Qwen3-30B-Instruct scored 48.1% (below chance level), indicating a strong tendency to reproduce bias when generating intuitively.
  • Qwen3-30B-Thinking (a model with a "thought process") scored 95.8%, suggesting that explicit reasoning steps help self-correct attribution errors.
• Discriminability: JUBAKU-v2 showed the highest score variance (0.0278) among all benchmarks, proving it is more sensitive to model differences than JBNLI (0.0137) or JBBQ (0.0008).
• Robustness:
  • GPT-4o and Qwen3-30B-Instruct showed high answer fluctuation rates (81.5% and 88.9%, respectively) when prompts were slightly altered.
  • GPT-5.2, Claude 4 Sonnet, and GPT-OSS-120B showed high stability with a fluctuation rate of only 7.4%.
• Correlation: JUBAKU-v2 correlated strongly with JBNLI but weakly with SSQA-JA, suggesting SSQA-JA suffers from ceiling effects where top models all score similarly, whereas JUBAKU-v2 reveals the hierarchy.

5. Significance and Implications

• Shift in Evaluation Paradigm: The paper argues that evaluating only the final output is insufficient. The reasoning process is where bias often hides, and fixing the conclusion allows for a pure measurement of reasoning bias.
• Importance of "Thinking" Processes: The stark contrast between the Instruct and Thinking versions of the Qwen3 model suggests that Chain-of-Thought (CoT) or inference steps are effective mechanisms for mitigating social bias, allowing models to override intuitive, biased associations from training data.
• Cultural Specificity: The study reinforces that bias evaluation must be culturally localized. A benchmark based on Western norms cannot adequately assess Japanese-specific stereotypes regarding gender, region, or social hierarchy.
• Future Directions: The authors plan to expand the dataset and extend this attribution-based reasoning evaluation framework to other languages.

In conclusion, JUBAKU-v2 provides a more rigorous, culturally relevant, and sensitive tool for assessing the fairness of LLMs, revealing that even state-of-the-art models harbor latent biases in their reasoning that are only exposed through controlled, reasoning-focused evaluation.