Do Language Models Follow Occam's Razor? An Evaluation of Parsimony in Inductive and Abductive Reasoning

1. Problem Statement

While Large Language Models (LLMs) have shown significant progress in deductive reasoning (deriving conclusions from fixed premises), their capabilities in non-deductive reasoning—specifically inductive (generalizing from cases) and abductive (inferring the best explanation for observations) reasoning—remain underexplored.

A critical aspect of non-deductive reasoning is Occam's Razor: the principle that among multiple valid hypotheses explaining a set of observations, the simplest one (the most parsimonious) is often the most useful and probable. Existing benchmarks often conflate reasoning with commonsense knowledge acquisition or focus solely on the correctness of the final answer (label accuracy) without evaluating the quality or simplicity of the reasoning process.

Core Research Question: Do LLMs possess the ability to perform inductive and abductive reasoning, and if so, do they adhere to Occam's Razor by generating the simplest, high-quality hypotheses?

2. Methodology

A. The INABHYD Dataset

The authors introduce INABHYD (Inductive and Abductive Hypothesis Discovery), a new benchmark dataset designed to evaluate these capabilities in a controlled setting.

• Synthetic Generation: To avoid training data contamination, the dataset uses fictional world models (e.g., fictional concepts like "wumpus" or "dalpist") rather than real-world entities.
• Structure: Each example consists of:
  1. World Model: An ontology tree represented in First-Order Logic (FOL), containing concepts, properties, members, and subtype relationships.
  2. Hidden Axioms: Specific axioms (ground truth) are hidden from the model.
  3. Observations: A set of facts derived from the world model and the hidden axioms.
  4. Task: The LLM must generate hypotheses to explain the observations under the incomplete world model.
• Difficulty Control: Complexity is controlled by the height of the ontology tree (1 to 4) and the number of required hypotheses (single vs. multiple).
• Scale: The dataset contains over 2,000 reasoning questions with generation code allowing for infinite expansion.

B. Evaluation Metrics

The paper proposes a novel framework to evaluate both correctness and parsimony:

1. Weak Accuracy: Measures if the generated hypotheses can explain all observations (logical validity).
2. Strong Accuracy: Measures if the generated hypotheses exactly match the ground-truth (simplest) hypotheses.
3. Hypothesis Quality (Parsimony Metric): A quantitative metric based on Occam's Razor.
   • It calculates the ratio of the average usage of predicted hypotheses in proof trees versus the average usage of ground-truth hypotheses.
   • $q(H) = \frac{1}{|H|} \sum n(h) / \frac{1}{|H^*|} \sum n(h^*)$
   • A score of 1.0 indicates the model found the most parsimonious set of hypotheses. Lower scores indicate unnecessary or overly complex hypotheses.

C. Experimental Setup

• Models Evaluated: Seven state-of-the-art LLMs, including GPT-5.4, O3, LLaMA3-70B, Gemma3-27B, DeepSeek-V3, GPT-4o, and DeepSeek-R1-Distill-LLaMA-70B.
• Techniques Tested:
  • Zero-shot prompting.
  • In-Context Learning (ICL) with 8-shot demonstrations (both in-distribution and out-of-distribution).
  • Reasoning-enhanced prompts (Chain-of-Thought vs. Buffer-of-Thought).
  • Reasoning models trained with Reinforcement Learning with Verifiable Rewards (RLVR).

3. Key Contributions

1. Framework for Non-Deductive Reasoning: A synthetic framework that simultaneously tests inductive and abductive reasoning using FOL expressible world models, separating reasoning ability from external knowledge.
2. Parsimony Metric: A new automated metric to quantitatively assess whether LLMs adhere to Occam's Razor, moving beyond simple binary correctness.
3. INABHYD Benchmark: A scalable dataset specifically designed to evaluate hypothesis discovery and quality, addressing the gap in existing literature which focuses heavily on deduction.
4. Systematic Evaluation: The first comprehensive study evaluating LLMs on the quality (simplicity) of inductive/abductive hypotheses, not just their validity.

4. Key Results

A. Performance Trends

• Simple Scenarios: LLMs perform well (high weak and strong accuracy >80%) on shallow ontology trees (height 1) with single hypotheses.
• Complex Scenarios: Performance degrades significantly as the ontology tree height increases (complexity) and when multiple hypotheses are required.
  • For height-4 trees, accuracy drops below 50% for most models.
  • Weak vs. Strong Accuracy Gap: Models frequently produce valid hypotheses (Weak Accuracy) that explain observations but fail to produce the simplest hypotheses (Strong Accuracy and Quality scores remain low).

B. Adherence to Occam's Razor

• Failure of Parsimony: Even when LLMs can explain the observations, they struggle to identify the simplest explanation. They often generate unnecessary hypotheses or trivial hypotheses (repeating observations as rules).
• Error Analysis: The most common error (Type 2) is ignoring the ontology structure and producing redundant hypotheses. Other errors include wrong ontology direction (e.g., inferring "All mammals are cats" instead of "All cats are mammals") and hallucinating entities.

C. Impact of Enhancements

• In-Context Learning (ICL): In-distribution demonstrations (matching the test difficulty) slightly improve performance, particularly for strong accuracy and quality in complex trees. Out-of-distribution demonstrations show negligible benefit.
• Reasoning Models & Prompts: Models like GPT-5.4 and O3 outperform base models. Techniques like Buffer-of-Thought (BoT) generally yield better results than standard Chain-of-Thought (CoT), especially for weak accuracy.
• Limitation: Despite these improvements, a persistent gap remains between weak and strong accuracy. Current techniques do not fully solve the problem of generating high-quality, parsimonious hypotheses in complex scenarios.

5. Significance and Conclusion

• Limitations of Current LLMs: The study reveals that while LLMs can perform basic non-deductive reasoning, they lack the inherent ability to apply Occam's Razor effectively in complex, multi-hypothesis scenarios. They tend to over-complicate explanations rather than finding the most parsimonious one.
• Implications for AI Safety and Science: Since inductive and abductive reasoning are crucial for scientific discovery and real-world problem solving, the inability to generate simple, high-quality hypotheses suggests current models may be unreliable for tasks requiring rigorous hypothesis generation.
• Future Directions: The authors suggest that standard reasoning enhancements (CoT, RLVR) are insufficient for this specific challenge. Future work may require specialized training on parsimony or neuro-symbolic approaches to enforce logical simplicity.

In summary, the paper argues that LLMs currently do not reliably follow Occam's Razor in complex inductive and abductive reasoning tasks, often producing valid but unnecessarily complex hypotheses, highlighting a significant gap between current model capabilities and human-level scientific reasoning.