CommonWhy: A Dataset for Evaluating Entity-Based Causal Commonsense Reasoning in Large Language Models

Technical Summary: CommonWhy

Problem Statement
Large Language Models (LLMs) have demonstrated proficiency in encoding factual knowledge and general commonsense, yet their ability to perform entity-based commonsense reasoning—specifically regarding causal relationships and constraints in the real world—remains inadequately evaluated. Existing benchmarks for entity-based reasoning (e.g., StrategyQA, CREAK) primarily rely on True/False or multiple-choice formats that test deductive verification or pattern matching. These datasets fail to explicitly assess a model's capacity for abductive reasoning (inference to the best explanation) when answering "why" questions. Furthermore, while Knowledge Graph Question Answering (KGQA) datasets exist, they predominantly focus on factoid retrieval or multi-hop fact chaining, lacking the requirement to integrate implicit commonsense assumptions with factual knowledge to generate causal explanations.

Methodology
The authors introduce CommonWhy, a dataset comprising 15,000 "why" questions designed to evaluate entity-based causal commonsense reasoning. The dataset construction follows a three-stage pipeline:

1. Axiom Generation: To overcome the limitations of extraction-based methods (which often yield questions answerable by simple retrieval), the authors leverage LLMs (GPT-5.1) to generate "lifted" commonsense axioms. Starting from a seed set of axioms extracted from existing datasets (StrategyQA, CREAK, CoLoTa), the LLM is prompted to creatively expand these into new causal structures. These generated axioms are rigorously filtered by human annotators to ensure they align with human commonsense.
2. Query Generation: The validated axioms are rewritten as abstract question-answer pairs. These are then grounded by instantiating abstract variables with concrete entities from the Wikidata knowledge graph. The authors use SPARQL queries to retrieve entities that satisfy the specific factual constraints of the axioms, ensuring all necessary factual knowledge is available within Wikidata. The resulting queries are rewritten into fluent, natural language "why" questions.
3. Multi-Answer Annotation: Recognizing that real-world causal effects often admit multiple valid explanations, the dataset adopts a multi-answer format. Human annotators inspect the relevant Wikidata subgraphs for each query to identify all plausible causal explanations supported by the facts and commonsense assumptions. Only causes agreed upon by multiple annotators are retained as ground truth.

The dataset is split into Head and Long-Tail subsets based on entity popularity (Wikidata triple counts) to evaluate model robustness across different levels of entity familiarity.

Key Contributions

• CommonWhy Dataset: The first large-scale dataset specifically designed to evaluate abductive, entity-based causal reasoning. It bridges the gap between commonsense reasoning and KGQA by requiring models to combine factual retrieval with implicit causal inference.
• New KGQA Paradigm: Unlike traditional KGQA benchmarks that test fact retrieval, CommonWhy establishes a benchmark for explanation-driven reasoning over knowledge graphs, where answers require synthesizing KG facts with commonsense axioms.
• Multi-Answer Ground Truth: The dataset explicitly models the non-unique nature of causal reasoning by providing multiple valid ground-truth answers for each query, moving beyond single-answer constraints.

Experimental Results
The authors evaluated state-of-the-art LLMs (including GPT-5.1, Gemini-2.5-Flash, Llama-3.3-70B, OpenAI-o3, and DeepSeek-V3.2) and LLM-based KGQA methods (KB-Binder, KGR) on CommonWhy. Key findings include:

• Reasoning Limitations: Even the strongest reasoning models struggle with the task. OpenAI-o3, the best-performing model, achieved only ~68% correctness on the Head split and ~67% on the Long-Tail split. Other models performed significantly worse (e.g., DeepSeek-V3.2 at ~41%).
• Metric Discrepancy: Standard long-form generation metrics (BERTScore, ROUGE-L, METEOR) were found to be poor indicators of causal correctness. Models like Gemini-2.5-Flash achieved high lexical similarity scores but low logical correctness, suggesting that surface-level fluency does not equate to valid abductive reasoning.
• Entity Popularity Impact: Performance consistently degraded when moving from Head to Long-Tail entities, confirming that entity popularity influences reasoning robustness even when the underlying reasoning patterns are identical.
• Hallucinations: Even when models produced logically correct answers, they frequently exhibited factual hallucinations (unsupported atomic claims), as measured by FActScore. This issue was more pronounced for Long-Tail entities.
• KGQA Inadequacy: Existing KGQA methods failed to adapt to this task. KB-Binder, which translates queries to SPARQL, performed poorly because SPARQL cannot capture the implicit commonsense required for "why" questions. KGR showed no significant improvement over direct prompting, indicating current pipelines are insufficient for abductive causal reasoning.

Significance
The paper claims that CommonWhy reveals a critical gap in current LLM capabilities: success on existing deductive or verification-based commonsense benchmarks does not transfer to abductive causal reasoning in entity-based settings. By establishing a benchmark where factual knowledge (Wikidata) is necessary but insufficient without commonsense inference, CommonWhy provides a challenging testbed for future research. It highlights the need for new methods capable of integrating factual retrieval with implicit causal reasoning to effectively interact with the real world, marking a shift in KGQA evaluation from fact retrieval to explanation-driven reasoning.