Data-driven Machine Learning Cannot Reach Symbolic-level Logical Reasoning -- The Limit of the Scaling Law

Technical Summary: Data-driven Machine Learning Cannot Reach Symbolic-level Logical Reasoning – The Limit of the Scaling Law

Problem Statement
The paper addresses a fundamental question regarding the limits of data-driven machine learning: Can supervised systems attain the rigour of symbolic-level logical reasoning (specifically Aristotelian syllogistic reasoning) solely by scaling training data and computation time? While recent advancements in Large Language Models (LLMs) and deep learning have demonstrated impressive capabilities in various domains, their ability to perform rigorous, out-of-distribution logical deduction remains unproven. The authors contrast these data-driven approaches with Sphere Neural Networks (SphNNs), which achieve symbolic-level reasoning through explicit model construction without training data, raising the question of whether scaling laws can bridge the gap between statistical pattern recognition and logical necessity.

Methodology
The paper employs a combination of theoretical analysis and empirical experimentation to investigate two primary paradigms of supervised syllogistic reasoning:

1. Image-Input Supervised Learning: The authors analyze "Euler Net," a convolutional deep learning network that treats syllogistic reasoning as an image recognition problem using Euler diagrams. They extend this to "Super Euler Net" (SupEN), an automated system capable of generating new training data to identify and correct reasoning errors.
2. Symbolic-Input Supervised Learning (LLMs): The study evaluates the latest generation of OpenAI models (GPT-5 and GPT-5-nano) on syllogistic reasoning tasks.
3. Experimental Design:
   • Data Exhaustion Tests: Experiments were conducted to determine if increasing training data could distinguish all 24 valid forms of Aristotelian syllogisms or exhaust "unintended" inputs (e.g., partial circles).
   • Surface Form Variations: LLMs were tested on four surface lexical patterns: meaningful words, double words, simple symbols, and long random symbols.
   • Evaluation Metrics: Beyond standard decision accuracy, the authors introduced a stricter metric for symbolic-level reasoning: the requirement for both a correct decision and a correct justification.

Key Contributions and Findings

• Methodological Obstacle 1: Indistinguishability of Valid Forms via Training Data
  The authors demonstrate that training data alone cannot distinguish all 24 valid forms of Aristotelian syllogistic reasoning. In image-based systems like Euler Net, the combination table maps premises to conclusions based on set relations. However, distinct valid syllogisms (e.g., Barbara vs. Barbari) can share identical premise configurations but require different conclusions. Statistical learning on these inputs results in a performance range of 50% to 100% across valid types, failing to achieve the determinacy required for symbolic logic.

• Methodological Obstacle 2: Contradictory Objectives in End-to-End Learning
  The paper identifies a fundamental conflict in end-to-end supervised architectures. The pattern recognition component (e.g., Siamese networks) is trained to recognize wholes from parts (a desirable feature in computer vision), whereas the logical reasoning component must strictly adhere to the information present in the premises without "hallucinating" new objects. The authors show that image-input networks often "complete" partial inputs (e.g., inferring a full circle from a half-circle) to satisfy pattern recognition, leading to logically invalid conclusions that the reasoning component cannot detect or correct.

• Empirical Results on Scaling and Generalization

  • SupEN Experiments: Even with an automated loop of generating new training data and retraining, SupEN's accuracy on randomly generated test data plateaued at approximately 97.8%, and on a dataset covering all 24 valid types, it reached only 76% overall accuracy. Crucially, the system could not exhaust unintended inputs; generating new training data to cover one type of unintended input inevitably created new, untrained unintended patterns.
  • LLM Experiments: While GPT-5 and GPT-5-nano achieved near-perfect decision accuracy (up to 100%) on benchmarks, they frequently produced incorrect or inconsistent explanations. For instance, GPT-5 achieved 100% decision accuracy on long random symbols but provided wrong justifications for 25% of those correct decisions. The authors argue that empirical training typically terminates once decision accuracy is maximized, removing the pressure to correct underlying reasoning defects.

• Latent Feature Analysis: Experiments revealed that Euler Net does not necessarily learn abstract geometric relations. When the color of input circles was altered, accuracy dropped significantly (to 8.1%), suggesting the network relies on specific pixel-level correlations rather than abstract spatial logic.

Significance and Claims
The paper argues that supervised machine learning systems, whether trained on symbolic or image-based inputs, cannot be guaranteed to attain the rigour of symbolic syllogistic reasoning through scaling alone. The authors posit that the "Scaling Law" has a fundamental boundary: increasing data and computation cannot resolve the methodological deficits where statistical regularities fail to capture logical necessity.

The significance of this work lies in its identification of a "fundamental boundary" for data-driven reasoning. It suggests that achieving sophisticated logical reasoning requires moving beyond the paradigm of purely statistical learning and training data, potentially necessitating explicit model construction or hybrid approaches (like SphNNs) that do not rely on the assumption that more data leads to infinite logical capability. The paper concludes that without addressing the conflict between pattern recognition and logical deduction, supervised systems will remain unable to guarantee the correctness of their reasoning processes, particularly for out-of-distribution inputs.