Human-Level Reasoning: A Comparative Study of Large Language Models on Logical and Abstract Reasoning

Technical Summary: Human-Level Reasoning? A Comparative Study of Large Language Models on Logical and Abstract Reasoning

Problem Statement
While Large Language Models (LLMs) have demonstrated significant proficiency in natural language processing and text generation, their capacity for genuine logical and abstract reasoning remains a critical area of inquiry. Existing literature suggests that while LLMs adeptly capture statistical patterns within language, they often exhibit inconsistencies in applying logical principles, particularly when tasks require establishing logical interconnections between concepts or deriving valid inferences from intricate premises. This study addresses the need to rigorously evaluate whether these models truly understand information and perform deductions in a logical, valid manner, or if they merely simulate reasoning through statistical association.

Methodology
The study employed a comparative assessment paradigm involving fifteen distinct LLMs (including GPT-4.1, Claude 3.7 Sonnet, DeepSeek R1/V3, Gemini 2.0/2.5, Llama 3.1/4, Grok 2/3, Mistral, Perplexity, and the Brazilian model Sabiá 3) and a cohort of 80 human participants (54 students and 26 professors in quantitative disciplines).

• Benchmark Design: Instead of utilizing standardized benchmarks like the Winograd Schema Challenge or ARC, the authors developed a bespoke set of eight logical and abstract reasoning challenges. These questions were designed to minimize reliance on specific factual knowledge, focusing instead on pattern recognition, deduction, and abstraction. The set included:
  • Deductive Reasoning: Operations involving days of the week (Q1, Q7), Caesar cipher encoding (Q3), and arithmetic with missing correct options (Q4).
  • Inductive Reasoning: Linguistic associations between Portuguese and Spanish month abbreviations (Q6).
  • Abductive Reasoning: Binary number operations requiring conversion to decimal (Q8).
  • Abstract Reasoning: Pattern recognition in word repetition sequences (Q2) and complex digit concatenation logic (Q5).
• Evaluation Protocol: All models and humans were presented with the questions in Portuguese. The LLMs were prompted to elucidate their reasoning processes. To ensure consistency, two instances of each LLM were evaluated, with the highest-performing instance selected for analysis.
• Scoring System: The study utilized a qualitative scoring approach rather than statistical metrics like Perplexity or BLEU. Responses were categorized and scored as follows:
  • 10 points: Correct answer.
  • 5 points: Incorrect answer but with valid reasoning/judgment.
  • 0 points: No answer or invalid reasoning.

Key Contributions

1. Custom Reasoning Benchmark: The introduction of a specific set of eight challenges designed to test logical and abstract reasoning without relying on pre-trained factual knowledge, offering greater control over the nature of the evaluation compared to existing standardized benchmarks.
2. Qualitative Reasoning Analysis: A shift from purely statistical evaluation to a qualitative analysis of the cognitive processes underlying model responses, distinguishing between correct answers, correct reasoning with incorrect answers, and complete failures.
3. Human-LLM Comparative Baseline: A direct performance comparison between state-of-the-art LLMs and human experts (professors) and students, highlighting specific domains where human intuition and abstract reasoning outperform current AI capabilities.

Results

• Overall Performance: The mean score for LLMs was 59.7 points, while the human cohort (aggregated) scored 69.6 points. However, the top-performing LLMs (GPT-4o-mini, GPT-3.5-mini, and Claude 3.7 Sonnet) achieved maximum scores (80 points), outperforming the average human score.
• Task-Specific Disparities:
  • Strengths: LLMs performed exceptionally well on Q4 (arithmetic with missing options), where 100% of models recognized the absence of a correct choice, and Q6 (linguistic association), where they achieved 100% accuracy. They also excelled in Q1 and Q3.
  • Weaknesses: LLMs struggled significantly with Q5 (complex abstract pattern involving digit concatenation), achieving only 60% accuracy compared to 36.2% for humans (though professors scored 65.4%). Notably, many LLMs attempted to fit the data into polynomial functions rather than identifying the concatenation logic.
  • Language and Pattern Nuances: Models showed difficulty with Q7, which required alternating between English and Portuguese based on the parity of the increment. Many models failed to identify the switching rule, even when they correctly calculated the day.
• Model Variance: Performance varied widely. While top models (GPT-4o-mini, Claude 3.7 Sonnet) scored 80, lower-performing models like Sabiá 3 (25 points) and Llama 3.1 (40 points) struggled with basic pattern recognition and language switching.
• Reasoning vs. Accuracy: In several instances, models provided incorrect final answers but demonstrated valid reasoning steps, or conversely, arrived at correct answers through coincidental or flawed logic (e.g., Gemini 2.0 Flash on Q7).

Significance and Claims
The paper posits that while LLMs have made significant strides in pattern recognition and natural language processing, a gap remains between artificial and human intelligence regarding higher-order cognitive functions. The study claims that LLMs frequently struggle with tasks requiring nuanced understanding, abstract thought, and the integration of disparate information, particularly when problem formats are unconventional or rely on hidden rules (such as language switching or digit concatenation).

The authors conclude that the ability of LLMs to "truly reason" is currently limited by their reliance on statistical correlations and a lack of genuine understanding. The findings suggest that while models can surmount certain logical constraints, they often fail to apply logical principles consistently in scenarios demanding fluid intelligence and common-sense reasoning. The study emphasizes that future development must address these limitations to create models capable of surmounting the constraints of statistical pattern matching to achieve robust logical inference.