BIS Reasoning 1.0: The First Large-Scale Japanese Benchmark for Belief-Inconsistent Syllogistic Reasoning

This paper introduces BIS Reasoning 1.0, the first large-scale Japanese benchmark for evaluating belief-inconsistent syllogistic reasoning in LLMs, demonstrating that robustness against belief bias is driven primarily by explicit reasoning optimization rather than language specialization or model scale alone.

The Gist

🧠 The Big Idea: Can AI Think, or Just Guess?

Imagine you are taking a logic test. The teacher gives you two rules:

1. Rule: All cats are dogs.
2. Rule: All dogs are purple.
3. Question: Therefore, are all cats purple?

If you are a human who knows cats aren't dogs, your brain screams, "Wait, that's wrong!" You might get confused and say "No" because your real-world knowledge is fighting the logic.

But if you are a perfect logic machine, you ignore the fact that cats aren't actually dogs. You follow the rules strictly: If A is B, and B is C, then A must be C. So, you correctly answer "Yes."

This paper is about building a giant test to see if AI (Large Language Models) can be that perfect logic machine, or if they get tripped up by their own "common sense."

---

🏗️ What Did They Build? (The "BIS" Dataset)

The researchers created a new dataset called BIS Reasoning 1.0. Think of this as a giant obstacle course for AI, specifically designed to trick them.

• The Trap: The test contains 5,000 puzzles in Japanese. In every puzzle, the answer is logically correct, but it sounds absurd or wrong based on real life.
  • Example: "All birds can fly. Penguins are birds. Therefore, penguins can fly." (Logically valid based on the rules, but factually false in the real world).
• The Goal: They wanted to see if the AI would say "No" because it knows penguins can't fly (believing bias), or "Yes" because it followed the math of the sentence (logical reasoning).

They call this "Belief-Inconsistent Reasoning." It's like asking the AI to wear "logic goggles" and ignore its "common sense glasses."

---

🏆 The Race: Who Passed the Test?

The researchers put many different AI models through this obstacle course. Here is how they fared:

1. The "Super-Reasoners" (The Gold Medalists) 🥇

• Who: Newer, specialized models like GPT-5 and Qwen.
• Performance: They got almost 100% correct.
• Analogy: These are like students who have been trained specifically to ignore distractions. When they see the puzzle, they put on their "logic goggles," ignore the fact that penguins can't fly, and solve the math perfectly.

2. The "Old Guard" (The Strugglers) 🥉

• Who: Older models like GPT-4o and early Japanese models.
• Performance: They scored around 60–80%.
• Analogy: These models are like smart people who are too easily distracted. When they see "Penguins can fly," their brain says, "Wait, that's impossible!" and they get confused, failing the logic test even though the answer was right based on the rules.

3. The "Japanese Specialists" (The Up-and-Comers) 🇯🇵

• Who: Models built specifically for the Japanese language (like llm-jp).
• Performance: The older versions were terrible (scoring below 50%, sometimes even guessing wrong on purpose!). But the newest version (llm-jp-3.1) jumped up to the 80s.
• Analogy: The old Japanese models were like chefs who knew how to cook delicious Japanese food but couldn't follow a recipe if the ingredients were weird. The new version learned to follow the recipe strictly, even if the ingredients were strange.

---

🔍 Key Discoveries (The "Aha!" Moments)

1. Size Doesn't Matter, Training Does 📏

You might think a bigger, smarter AI would automatically be better at logic. Not true.

• Analogy: A giant library full of books (a big model) doesn't mean the librarian knows how to solve a riddle. It's about how the librarian was trained. If you train them to prioritize logic over facts, they get better. If you just train them to sound polite or fluent, they get worse at logic.

2. The "Prompt" is the Secret Sauce 📝

The researchers found that how you ask the question changes the answer.

• The "Casual" Ask: "Hey, does this make sense?" -> The AI gets lazy and uses common sense.
• The "Strict" Ask: "Ignore real life. Follow the rules step-by-step." -> The AI wakes up and solves it correctly.
• Analogy: It's like asking a friend, "What's 2+2?" vs. "Pretend 2+2 equals 5. Now, what is 2+2?" The second question forces them to switch gears.

3. The "Thinking Time" Matters ⏳

Newer models have a setting called "Reasoning Effort."

• Low Effort: The AI guesses quickly. (Score: Low)
• High Effort: The AI takes a moment to "think" through the steps. (Score: High)
• Analogy: It's the difference between a student who guesses the answer on a multiple-choice test in 2 seconds vs. one who actually works out the problem on scratch paper.

---

🚨 Why Should We Care? (The Real World Impact)

Why do we need to test AI on "nonsense" logic? Because real life is full of traps.

Imagine an AI lawyer or a doctor:

• The Lawyer: If a client says, "All people who wear red are guilty. John wears red. Therefore, John is guilty," a biased AI might say, "Well, John looks innocent, so no." But a logical AI must say, "Based on the rule you gave me, yes, he is guilty."
• The Doctor: If a new study says, "All patients with this symptom have a rare disease," the AI shouldn't say, "That can't be true, I've never heard of it." It needs to follow the evidence, even if it feels weird.

The Conclusion:
To make AI safe for law, medicine, and science, we can't just rely on them being "fluent" or "polite." We need to train them to be stubbornly logical, even when the answer feels wrong. This new test (BIS Reasoning 1.0) is the first step in making sure Japanese AI can do that.

Technical Summary

1. Problem Statement

Large Language Models (LLMs) have demonstrated impressive fluency but struggle with reliable logical reasoning, particularly when logical validity conflicts with human intuition or prior knowledge. This phenomenon is known as belief bias: the tendency to accept conclusions that align with real-world beliefs even if they are logically invalid, or to reject valid conclusions that contradict common sense.

While significant progress has been made in English-language logical reasoning benchmarks (e.g., ReClor, LogiQA), there is a critical gap in Japanese-language evaluation. Existing Japanese datasets like JFLD focus on formal logic isolated from real-world knowledge (using unnatural synthetic sentences), and NeuBAROCO covers belief bias but lacks scale and a dedicated focus on belief-inconsistent reasoning. Consequently, there is no large-scale, naturalistic benchmark to evaluate how well Japanese LLMs can prioritize formal logic over intuitive beliefs.

2. Methodology

Dataset Construction: BIS Reasoning 1.0

The authors introduced BIS Reasoning 1.0, a dataset of 5,000 carefully curated syllogistic reasoning problems in Japanese.

• Design: Each item consists of two premises and a conclusion that is logically valid (strictly entailed by syllogistic rules) but belief-inconsistent (contradicts common sense or factual knowledge).
• Categorization: The dataset covers 46 raw semantic categories (e.g., Animals, Healthcare, Law, Technology), consolidated into 10 final broad categories (e.g., Animals & Living Things, Human Body, Logic & Structure) to ensure diversity and balance.
• Quality Control: A two-phase QA process involving native Japanese speakers ensured linguistic fluency, structural validity, and the absence of ambiguity.

Experimental Setup

• Models Evaluated: A comprehensive suite of models was tested under a uniform zero-shot protocol using Japanese prompts.
  • General-purpose: OpenAI GPT-5 variants (mini, nano, 4o, 4-turbo), gpt-oss-20b, Qwen3-32B.
  • Japanese-specialized: llm-jp-3 variants (13b, instruct3, instruct4), Stockmark-13b.
  • Legacy/Deprecated: Claude-3 Sonnet/Opus (early 2024 versions).
• Task Formulation: Models were asked to determine if a conclusion logically follows from premises, answering "Yes" or "No."
• Metrics: Accuracy was measured as the proportion of correct "Yes" judgments (since all BIS entries are logically valid).
• Prompt Variations: The study analyzed the impact of different prompt styles (Basic, Focus Logic, Chain-of-Thought, Polite, Casual) and inference-time reasoning efforts (e.g., GPT-5 "medium" vs. "minimum" effort).

3. Key Contributions

1. BIS Reasoning 1.0 Dataset: The first large-scale, naturalistic Japanese dataset explicitly designed to test belief-inconsistent syllogistic reasoning, filling a major gap in non-English logical evaluation.
2. Comprehensive Benchmarking: The first systematic comparison of state-of-the-art general and Japanese-specialized LLMs on belief-inconsistent reasoning tasks.
3. Analysis of Bias and Reasoning: Detailed quantification of belief bias across model generations, revealing that reasoning optimization is a stronger driver of performance than language specialization or scale alone.
4. Prompt Sensitivity Insights: Demonstration that prompt design (specifically Chain-of-Thought and explicit logical framing) significantly mitigates belief bias in models that otherwise struggle.

4. Key Results

Overall Performance

• Reasoning-Optimized Models: Models explicitly optimized for reasoning (GPT-5 variants, Qwen3-32B) achieved near-perfect accuracy (~99% to 99.7%).
• Japanese-Specialized Models:
  • Early Generations: Older models (e.g., llm-jp-3-13b-instruct3) performed poorly, often below 11%, indicating a systematic tendency to favor belief-consistent (but invalid) conclusions.
  • Latest Generation: The newest model, llm-jp-3.1-13b-instruct4, showed a marked improvement to ~84.7%, suggesting a successful integration of reasoning-aligned fine-tuning.
• Legacy Models: Early 2024 Claude models performed drastically worse on BIS Reasoning 1.0 (7–20%) compared to NeuBAROCO, highlighting the specific difficulty of the belief-inconsistent task.

Impact of Reasoning Effort and Prompts

• Reasoning Effort: For GPT-5-nano, accuracy dropped from 98.8% (medium effort) to 69.2% (minimum effort), proving that inference-time reasoning capacity is crucial for overcoming belief bias.
• Prompt Engineering: In error analysis of GPT-4o:
  • Chain-of-Thought (CoT): Improved accuracy on failed samples to 87%.
  • Focus Logic: Explicitly mentioning belief inconsistency improved accuracy to 76%.
  • Casual/Basic Prompts: Resulted in very low recovery rates (3–5%), showing models default to heuristics without strict logical framing.

Category Analysis

• Top-tier models showed consistent performance across all 10 categories.
• Lower-performing models (e.g., GPT-4o, GPT-4-turbo) showed specific weaknesses in the "Animals & Living Things" category, likely due to training data distributions.

5. Significance and Implications

• Distinction Between Fluency and Logic: The results confirm that linguistic fluency and instruction adherence (hallmarks of many Japanese LLMs) do not guarantee logical robustness. Explicit reasoning optimization is required.
• Safety-Critical Applications: The study highlights risks in deploying LLMs in law, healthcare, and scientific research, where models must override intuitive beliefs to maintain logical fidelity.
• Future Directions: The paper advocates for "reasoning-centric" fine-tuning strategies in Japanese LLM development, moving beyond mere instruction following to prioritize logical consistency.
• Benchmarking Standard: BIS Reasoning 1.0 provides a necessary diagnostic tool to expose latent cognitive biases in LLMs that standard benchmarks often overlook.

In conclusion, BIS Reasoning 1.0 demonstrates that while the latest Japanese LLMs are beginning to close the gap with global leaders through reasoning-focused training, belief bias remains a fundamental challenge that requires specific architectural and training interventions to overcome.