🤖 AI

Bongards at the Boundary of Perception and Reasoning: Programs or Language?

This paper introduces a neurosymbolic approach that combines Large Language Models for generating parameterized programs with Bayesian optimization for parameter fitting to solve Bongard problems, thereby bridging the gap between visual perception and abstract reasoning.

Original authors: Cassidy Langenfeld, Claas Beger, Gloria Geng, Wasu Top Piriyakulkij, Keya Hu, Yewen Pu, Kevin Ellis

Published 2026-02-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Cassidy Langenfeld, Claas Beger, Gloria Geng, Wasu Top Piriyakulkij, Keya Hu, Yewen Pu, Kevin Ellis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Visual Riddle" Challenge

Imagine you are given a set of 12 drawings. Six of them are "good" (positive), and six are "bad" (negative). Your job is to figure out the secret rule that separates the good ones from the bad ones.

This is called a Bongard Problem. These aren't just simple "find the odd one out" puzzles. The rules can be incredibly tricky.

Example: Maybe the "good" pictures all have a shape with a "neck" that is horizontal, while the "bad" ones have a vertical neck. Or maybe the "good" ones have shapes that are "squiggly," while the "bad" ones are "smooth."

The paper asks a big question: Can Artificial Intelligence (AI) solve these riddles the way humans do? Humans are great at looking at a brand-new situation, inventing a new concept on the fly (like "horizontal neck"), and applying it. Current AI is great at recognizing things it has seen before (like "this is a cat"), but it often struggles when it has to invent a new concept from scratch.

The Problem: AI is Too Rigid or Too Vague

The researchers found that AI has two main ways of thinking, and both have flaws when solving these riddles:

The "Chatbot" Approach (Natural Language):
- How it works: The AI looks at the pictures and tries to describe the rule in words, like a detective writing a report.
- The Flaw: It's good at big ideas but bad at precision. It might say, "The good shapes are kind of pointy," but it can't tell the difference between a "slightly pointy" shape and a "very pointy" shape. It's like a chef who knows the recipe says "add a pinch of salt" but doesn't know exactly how much that is.
The "Programmer" Approach (Code):
- How it works: The AI writes a computer program to check the images. It can measure exact distances, angles, and counts.
- The Flaw: It's great at precision but bad at creativity. If the rule is "the shape looks like a sad face," a computer program struggles to define "sadness" mathematically. It's like a robot that can measure a room to the millimeter but can't understand the concept of "cozy."

The Solution: The "Translator" Team

The authors built a new system that acts like a team of two specialists working together. They call this a "neurosymbolic" approach (combining neural networks/AI with symbolic logic/programs).

Here is how their team works, step-by-step:

Step 1: The Idea Generator (The "Chatbot")

First, they ask a powerful AI (a Vision-Language Model) to look at the riddle and guess a few possible rules in plain English.

Analogy: Imagine a brainstorming session where a creative writer suggests, "Maybe the rule is about the shape's 'neck'!"

Step 2: The Translator (The "Programmer")

Next, the system takes those English guesses and tries to turn them into computer code.

The Twist: The AI doesn't just write the code; it leaves "blanks" for the tricky numbers. For example, if the rule is "shapes with a neck longer than X," the AI writes the code but leaves X as a mystery variable.
Analogy: The writer says, "The neck must be longer than [BLANK]." The programmer sets up the machine to measure the neck but waits for the exact number.

Step 3: The Tuner (Bayesian Optimization)

This is the secret sauce. The system runs a "trial and error" process to find the perfect number for those blanks. It tests different numbers (like 5 pixels, 10 pixels, 15 pixels) to see which one perfectly separates the "good" pictures from the "bad" ones.

Analogy: Imagine you are tuning a radio. You know the station is somewhere between 90 and 100, but you don't know the exact frequency. You slowly turn the dial until the static clears and the music is perfect. The system does this mathematically to find the exact "cutoff" point for the rule.

Step 4: The Judge (The Verifier)

Finally, the system checks: "Does this code actually work on all the training pictures?"

If the code works perfectly, the system accepts the rule.
If the code fails, the system goes back to the "Chatbot," says, "That idea didn't work, try again," and the cycle repeats.

What Did They Find?

The researchers tested this "Team Approach" against the best AI models available (like GPT-4o and Claude 3.7) and even against average human performance.

The Team Wins: By combining the creativity of the Chatbot with the precision of the Programmer, their system solved 51 out of 100 Bongard problems.
Beating Humans: The average human in previous studies solved about 47 problems. Their AI team solved 51, marking the first time an AI has surpassed average human performance on this specific test.
Different Strengths:
- When the problem was about geometry and math (like "are these lines parallel?"), the "Programmer" part of the team was the hero.
- When the problem was about abstract concepts (like "is this shape 'open' or 'closed'?"), the "Chatbot" part was the hero.
- When they used only the Chatbot or only the Programmer, they did worse. They needed both.

The "Memorization" Check

The researchers were worried that the AI might just be "cheating" by memorizing the answers from the internet (since these puzzles are old and popular). To test this, they flipped the rules: they told the AI that the "bad" pictures were actually the "good" ones. The AI still performed well, proving it wasn't just reciting memorized answers; it was actually figuring out the logic.

The Bottom Line

This paper shows that to solve hard visual puzzles, AI shouldn't just try to "think" like a human or "calculate" like a computer. It needs to do both. By letting a creative AI suggest ideas and a precise computer verify them with exact math, we can build systems that learn new visual concepts almost as well as, and sometimes better than, humans.

Technical Summary: Bongards at the Boundary of Perception and Reasoning: Programs or Language?

Problem Definition

The paper addresses the challenge of solving Bongard Problems (BPs), a classic set of 100 few-shot visual reasoning puzzles introduced in 1970. Each problem consists of 12 images: six representing a "positive" concept and six representing a "negative" concept. Unlike tasks such as Raven's Progressive Matrices, which rely on fixed logical sequences, BPs require the solver to perceive novel visual features on-the-fly and fluidly define new perceptual primitives to distinguish between the two categories. The core difficulty lies in the boundary between perception (identifying features like "neck," "concavity," or "circularity") and reasoning (formulating a rule that separates the classes). While Vision-Language Models (VLMs) excel at everyday visual tasks, they struggle with the radical generalization required by BPs, often failing to synthesize the precise geometric or conceptual rules needed for a solution.

Methodology

The authors propose a neurosymbolic approach that combines the high-level conceptual reasoning of Large Language Models (LLMs) with the precision of programmatic execution. The system operates through two primary tasks: Verification (classifying a held-out image given a ground-truth rule) and Solution (generating the rule from scratch).

System Architecture

The system comprises two main components: a Hypothesis Generator and a Verifier.

Hypothesis Generator:
- Utilizes a VLM to generate candidate natural language rules.
- To improve diversity and relevance, the generator uses Retrieval-Augmented Generation (RAG). It samples rules based on in-context examples drawn from previous Bongard problems (leveraging the dataset's natural curriculum) and random problems.
- For the Solution task, the generator produces multiple candidate rules; for the Verification task, it takes the ground-truth rule as input.
Verifier:
- The verifier evaluates candidate rules using a hybrid strategy that alternates between Program Induction and Transduction (Chain-of-Thought).
- Program Induction: For a given rule, the VLM attempts to synthesize a parameterized Python program (specifically a classify_image method) that implements the rule.
  - Parameterized Programs: Recognizing that exact numerical thresholds (e.g., defining "elongated" vs. "compact") are difficult for LLMs to guess, the system treats certain values as parameters.
  - Bayesian Optimization: The system performs 15 steps of Bayesian optimization to find the parameter values that maximize the program's accuracy on the training examples.
  - Scoring: A program is accepted if it achieves a score $\ge 0.9$ on the training set. If multiple programs succeed, the majority label on the test image determines the output.
- Transduction (CoT): If no program achieves the required score, the verifier falls back to a Chain-of-Thought approach, prompting the VLM to directly classify the test image based on the training examples and the proposed rule.

Experimental Setup

Models: The study evaluates GPT-4o and Claude 3.7 Sonnet.
Tasks:
- Verification: Classifying images given the ground-truth rule.
- Solution: Generating the correct natural language rule for the entire problem set.
- Inversion: A control task where positive and negative examples are swapped to test for memorization.
Baselines: Performance is compared against base VLMs (without program synthesis) and previous human performance data (Wüst et al., 2025).

Key Contributions

Evaluation of State-of-the-Art VLMs: The paper provides a rigorous benchmark of GPT-4o and Claude 3.7 on Bongard Problems, distinguishing between their ability to verify rules and generate them from scratch.
Hybrid Neurosymbolic Model: The authors introduce a novel model that synthesizes parameterized programs alongside natural language reasoning. This approach leverages the VLM's ability to generate high-level concepts and the program's ability to perform exact geometric calculations and parameter fitting.
Human-Model Comparison: The study analyzes human data to compare the strengths and weaknesses of VLMs and the proposed hybrid model against human behavior, identifying specific categories where models succeed or fail relative to humans.

Results

Verification Performance:
- Claude 3.7 demonstrated strong performance, achieving an accuracy of 86.5% when combining programmatic verification and CoT reasoning.
- The hybrid approach ("+ both") outperformed using programs alone or CoT alone, particularly in Spatial and Concept categories. Programmatic verification excelled at spatial and similarity-based problems, while CoT was superior for high-level conceptual and number-based problems.
- GPT-4o showed a larger gap between programmatic and CoT performance, with CoT generally outperforming program synthesis, suggesting that the effectiveness of the hybrid approach depends on the underlying model's code generation capabilities.
Solution Performance:
- The full system using Claude 3.7 as the hypothesis generator solved 51 out of 100 problems.
- This result exceeds the average human performance reported by Wüst et al. (2025), which was approximately 47 correct solutions.
- GPT-4o solved 31 problems with the hybrid method, an improvement over its baseline but still below human averages.
Memorization Check: The "Inversion" task showed that swapping positive and negative labels had little impact on model performance, suggesting that the models are not simply memorizing specific problem solutions but are engaging in genuine reasoning.

Significance and Claims

The paper claims that combining natural language and formal programs allows AI systems to reason more accurately about visual concepts than either medium alone. By generating concepts in language and implementing them in code, the system bridges the gap between the "perception" (identifying features) and "reasoning" (applying logic) required by Bongard Problems.

The authors state that their system, for the first time to their knowledge, has exceeded average human accuracy on the Bongard Problems. They frame this not as a final solution to general visual reasoning, but as a step toward AI systems that can "acquire and use new perceptual concepts in humanlike ways." The work highlights that while VLMs possess vast pretraining knowledge, they require the precision of programmatic execution to handle the geometric and parametric nuances of abstract visual puzzles. The study concludes that the complementary nature of induction (program synthesis) and transduction (CoT) is essential for tackling difficult concept learning tasks at the boundary of perception and reasoning.