Technical Summary: Do AI Models Perform Human-like Abstract Reasoning Across Modalities?

Problem Statement

While OpenAI's o3-preview model recently exceeded human accuracy on the Abstraction and Reasoning Corpus (ARC) benchmark, it remains unclear whether these models achieve the human-like abstract reasoning capabilities the benchmark was designed to test. High accuracy on ARC tasks could result from exploiting "shortcuts"—unintended correlations in the data—rather than grasping the generalizable, core-knowledge priors (e.g., objectness, spatial relations) intended by the benchmark designers. Furthermore, existing evaluations have relied almost exclusively on textual representations of grid tasks, leaving the performance of state-of-the-art multimodal models in the visual modality unexplored. This paper investigates whether AI models recognize and reason with the specific abstractions intended by the benchmark designers, or if they rely on superficial patterns, across both textual and visual modalities.

Methodology

The authors evaluated several commercial and open-weight models (OpenAI's o3, o4-mini, Google's Gemini 2.5 Pro, Anthropic's Claude Sonnet 4, and non-reasoning baselines) on the ConceptARC benchmark. ConceptARC consists of 480 tasks organized around 16 basic spatial and semantic concepts (e.g., "inside vs. outside," "top vs. bottom"). Unlike the more complex ARC, ConceptARC tasks are designed to be simple for humans, isolating specific abstract concepts to facilitate the analysis of reasoning strategies.

The study employed a multi-faceted evaluation framework:

1. Modalities: Models were tested in both textual (integer matrix representation) and visual (image input) modalities.
2. Reasoning Effort & Tools: Experiments varied reasoning effort (low vs. medium token budgets) and access to external Python tools (enabled vs. disabled).
3. Dual-Output Evaluation: For each task, models were required to generate:
   • The output grid (represented as an integer matrix).
   • A natural-language rule describing the transformation.
4. Rule Classification: Generated rules were manually classified into three categories:
   • Correct-Intended: Rules aligning with the benchmark's designed abstractions (human-like core knowledge).
   • Correct-Unintended: Rules that work on the demonstrations and produce correct outputs but rely on spurious patterns (e.g., numerical relationships between color codes) rather than intended concepts.
   • Incorrect: Rules that fail to describe the transformation or produce incorrect outputs.
5. Human Baseline: Human performance was evaluated using unpublished data from Moskvichev et al. (2023), where participants provided both output grids and rules.

Key Results

1. Accuracy vs. Reasoning Quality

• Textual Modality: In the textual setting, the best models (o3 with medium effort) surpassed human accuracy (73%) on ConceptARC, achieving ~77% accuracy. However, accuracy alone was misleading: approximately 27% of o3's correct outputs in the textual setting were based on correct-unintended or incorrect rules. In contrast, only about 8% of human correct outputs relied on such shortcuts.
• Visual Modality: Performance dropped sharply in the visual modality. Even with Python tools enabled, the best models achieved only ~29% accuracy (o3 medium + tools), significantly lagging behind humans.
• Rule-Grid Alignment: Despite low visual accuracy, a substantial share of models' generated rules in the visual modality were correct-intended. This indicates that models often successfully identified the abstract concept but failed to apply it correctly due to perceptual limitations (e.g., misidentifying grid dimensions or object placement).

2. Impact of Tools and Reasoning Effort

• Textual: Increased reasoning effort improved accuracy and rule correctness. Python tools provided marginal benefits in this modality.
• Visual: Increased reasoning effort alone had little effect. However, enabling Python tools (specifically computer vision libraries) resulted in substantial improvements in both output accuracy and rule correctness, suggesting that perceptual bottlenecks, rather than reasoning deficits, were the primary cause of failure in the visual modality.

3. Nature of Shortcuts

Models frequently relied on unintended features, such as:

• Numerical relationships between the integer codes assigned to colors (e.g., sorting shapes by color value rather than spatial position).
• Spurious patterns in the demonstration grids that did not generalize to the test grid's underlying concept.
• Focus on low-level pixel features rather than "objectness."

Key Contributions

1. Beyond Accuracy: The paper demonstrates that evaluating abstract reasoning solely via output accuracy substantially overestimates AI capabilities in textual modalities (by ignoring reliance on shortcuts) and underestimates them in visual modalities (by ignoring the presence of correct abstract reasoning despite perceptual failures).
2. Rule-Level Analysis: By introducing a manual evaluation of natural-language rules, the study reveals a significant gap between human and AI reasoning strategies. AI models are much more likely to solve tasks via superficial correlations than humans, even when achieving high accuracy.
3. Modality Discrepancy: The study highlights a critical disconnect in multimodal models: they can often formulate the correct abstract rule for visual tasks but fail to execute the transformation due to perceptual errors.

Significance and Claims

The authors claim that their findings offer a more faithful picture of AI models' abstract reasoning abilities. They argue that current benchmarks and evaluation metrics, which prioritize output accuracy, fail to capture whether models are truly reasoning by analogy using human-like priors.

The paper posits that for AI systems to achieve human-like generalization and explainability, they must move beyond "shortcuts" and robustly grasp the intended abstractions. The results suggest that while current models show promise in identifying abstract concepts, they lack the robustness to apply these concepts consistently, particularly in visual domains. The authors conclude that future progress toward human-like, abstraction-centered intelligence requires evaluation methods that assess the process (the rules generated) alongside the product (the output grid), and potentially necessitates architectures or training methods that explicitly incorporate human-like priors.