LOGICAL-COMMONSENSEQA: A Benchmark for Logical Commonsense Reasoning

1. Problem Statement

Current benchmarks for commonsense reasoning (e.g., CommonsenseQA, SocialIQA) typically rely on single-answer multiple-choice questions (MCQs). This format forces models to select one "gold" answer, obscuring the inherent ambiguity and complexity of real-world reasoning. In reality, many situations admit multiple plausible interpretations that can be:

• Jointly plausible (both are valid).
• Partially plausible (at least one is valid).
• Jointly implausible (neither is valid).

Existing benchmarks fail to evaluate whether Large Language Models (LLMs) can reason about the logical composition of these plausible statements. Consequently, models may appear competent by selecting a single correct answer while failing to understand the relationships between alternatives or handle negation-based reasoning.

2. Methodology: LOGICAL-COMMONSENSEQA

The authors introduce LOGICAL-COMMONSENSEQA, a benchmark that reframes commonsense reasoning as a logical composition task over pairs of atomic statements.

Dataset Construction Pipeline

The dataset is built by extending CommonsenseQA (12,247 MCQs) through a three-stage pipeline:

1. Generation: Using GPT-4o-mini, the authors over-generate 5,000 instances to create diverse sets of atomic answer candidates (both plausible and implausible) for each question.
2. Refinement & Pruning: The model filters candidates to remove trivial options, factually incorrect answers, and logically inconsistent pairs. This results in a curated set of 3 correct and 4 incorrect atomic options per question, requiring multi-step inference.
3. Logical Composition: A symbolic program deterministically combines pairs of atomic options using three plausibility-level operators:
   • AND: Both statements are independently plausible.
   • OR: At least one statement is plausible.
   • NEITHER/NOR: Neither statement is plausible.
   • MIXED: A setting where different operators appear across the answer choices within a single question to prevent pattern exploitation.

The final dataset contains 19,996 instances (4,999 for each operator type plus a MIXED set).

Human Validation

To ensure social grounding, the authors employed an awareness–consensus framework (Whiting and Watts, 2024). Two annotators evaluated options based on:

• Awareness: Do they personally believe the option is correct?
• Consensus: Do they believe most others would agree?
  This process yielded high inter-annotator agreement (AC2 > 0.84), confirming the dataset reflects socially grounded plausibility rather than just objective truth.

3. Key Contributions

• Novel Benchmark: LOGICAL-COMMONSENSEQA is the first benchmark to explicitly evaluate compositional commonsense reasoning using logical operators (AND, OR, NEITHER/NOR) over plausible alternatives, moving beyond single-answer prediction.
• Controlled Ambiguity: It preserves the standard MCQ format while introducing controlled ambiguity, forcing models to reason about the interaction between statements rather than just retrieving a single fact.
• Comprehensive Evaluation: The study evaluates a wide range of models (encoder-only, encoder-decoder, decoder-only, open-weight, and proprietary) under zero-shot, few-shot, and Chain-of-Thought (CoT) prompting, as well as supervised fine-tuning.

4. Experimental Results

The authors evaluated models including LLaMA-3, Qwen2.5, Gemini, and fine-tuned models (Flan-T5, DeBERTa).

Key Findings:

1. Performance on AND/OR: Models perform reasonably well on conjunctive (AND) reasoning (e.g., 70–80% F1) and moderately on disjunctive (OR) reasoning.
2. Failure on Negation: Performance collapses sharply on NEITHER/NOR (negation-based) questions.
   • Zero-shot LLMs (e.g., LLaMA-3-8B) achieve only ~13% F1 on NEITHER/NOR.
   • Even strong models like Gemini-3-Flash-Preview drop to ~39% on this specific operator.
3. MIXED Condition: Performance degrades further in the MIXED setting, where models must implicitly infer the governing operator for each option, suggesting models rely on surface-level heuristics rather than robust operator grounding.
4. Fine-Tuning vs. Zero-Shot:
   • Zero/Few-Shot: Models fail significantly on negation, indicating a fundamental reasoning limitation in inference-time.
   • Fine-Tuned: Supervised fine-tuned models achieve 83–95% F1 across all operators. This suggests the task is learnable with supervision, and the zero-shot failures are due to inference-time limitations rather than dataset artifacts.
5. Decomposition Analysis: The difficulty arises from the interaction of three factors:
   • Atomic plausibility estimation (models are decent at this).
   • Operator grounding (models struggle to apply the logic correctly).
   • Distractor competition (performance drops significantly when competing composite options are introduced, especially for NEITHER/NOR).

Comparison with CommonsenseQA

A model (LLaMA-3.1-8B) achieving 72.2% on standard CommonsenseQA drops to 13.9% on the NEITHER/NOR subset of LOGICAL-COMMONSENSEQA, demonstrating that traditional benchmarks significantly overestimate a model's true reasoning capabilities.

5. Significance and Implications

• Exposing Brittleness: The benchmark reveals that current state-of-the-art LLMs are brittle when faced with negation-based compositional reasoning. They tend to rely on "plausibility dominance" (selecting the most plausible statement regardless of the operator) rather than logical composition.
• Social Grounding: By validating data through an awareness-consensus framework, the benchmark highlights that commonsense is a distribution of shared beliefs, not just a set of facts.
• Future Directions: The authors suggest that future research must move beyond single-answer retrieval to compositional inference. The benchmark provides a controlled framework to test improvements in logical consistency, negation handling, and the ability to reason about multiple valid interpretations simultaneously.

In conclusion, LOGICAL-COMMONSENSEQA serves as a critical diagnostic tool, proving that while LLMs can retrieve commonsense facts, they struggle to logically compose and negate these facts in a way that mirrors human reasoning.