Technical Summary: ZendoWorld: Challenging AI Agents on Active Visual Concept Induction

Problem Statement

The central challenge addressed by this work is the ability of intelligent agents to jointly perceive complex visual inputs, form hypotheses about hidden logical patterns, and design informative experiments to test those hypotheses. While recent AI progress has focused on inductive reasoning from few-shot examples (e.g., ARC-AGI, Bongard problems) and active experimentation in symbolic domains, existing benchmarks largely treat induction as a passive, one-off task or rely on clean symbolic abstractions that lack the complexity of visual grounding. There is a critical gap in evaluating agents that must operate within a closed loop of perception (interpreting raw images), induction (inferring hidden rules), and experimentation (proposing new data points to reduce uncertainty) within a realistic visual environment.

Methodology: ZendoWorld

The authors introduce ZendoWorld, a controlled interactive benchmark inspired by the inductive logic game Zendo. It integrates perception, induction, and experimentation into a single evaluation loop.

Environment and Game Setup

• Visual Input: Scenes consist of 1–7 colored geometric objects (blocks, wedges, pyramids) with specific attributes (color, shape, orientation) and spatial relations (touching, stacked, pointing). Images are procedurally generated via a Blender–Prolog pipeline, yielding a combinatorial space of $\approx 10^{18}$ configurations.
• Hidden Rule: Each episode is defined by a latent ground-truth rule $r^*$ drawn from a Prolog-based Domain-Specific Language (DSL). Rules range from First-Order (counting single attributes) to Second-Order (parity, comparisons) and Compositional (spatial relations + logic).
• Interaction Loop:
  1. Initialization: The agent receives one positive and one negative seed example.
  2. Experimentation: The agent proposes a new scene (structured specification) and predicts its label. The environment returns the ground-truth label.
  3. Hypothesis Induction: If the prediction is correct, the agent may submit a rule hypothesis. If correct, the game ends; if incorrect, a counterexample is revealed, and the agent returns to the experimentation phase.
  4. Termination: The episode ends upon correct rule identification or exhaustion of a 30-example budget.

Evaluated Agents

The study evaluates four distinct agent architectures to isolate bottlenecks in perception, induction, and experimentation:

1. Oracle Agent (Neuro-Symbolic Upper Bound): Uses a pretrained vision model for perfect perception and a program synthesizer (DeepSynth) over the full DSL. It employs heuristic experiment selection for maximal disambiguation.
2. VLM Agent (End-to-End Neural): A Vision-Language Model (GPT-5-mini) operating without DSL access or explicit hypothesis search. It relies on prompting for both rule induction and experiment selection.
3. Bayesian Agent (LLM-SMC): Uses an LLM to propose natural language hypotheses and maintains a particle-based posterior via Sequential Monte Carlo (SMC) to guide experiment selection.
4. Vision-Language Programs (VLP) Agent: Combines VLM-based concept extraction (learning object/property vocabularies from images) with program synthesis over a general DSL.

Evaluation Protocol

• Metrics: Win rate (logical equivalence to $r^*$), average turns, label accuracy, structural F1 score of intermediate hypotheses, and Expected Information Gain (EIG) of proposed experiments.
• Human Baseline: A study with 19 participants playing the same protocol to benchmark human inductive reasoning capabilities.
• Ablations: Experiments were conducted in a symbolic-input setting (replacing images with Prolog descriptions) to disentangle perception failures from induction failures.

Key Results

1. Label Accuracy Does Not Imply Rule Recovery

A primary finding is that high accuracy in predicting labels for observed examples does not guarantee the recovery of the underlying rule. The Bayesian Agent achieved the highest label accuracy (75.2%) among non-oracle agents but solved only 13.6% of games. Conversely, the VLM Agent solved 44.5% of games but with lower label accuracy (56.7%). This dissociation suggests that correct classification can be achieved through pattern matching without understanding the generative rule.

2. Perception and Induction are Distinct Bottlenecks

• Perception: Removing visual input (symbolic setting) significantly improved the performance of the VLP Agent (win rate increased by 166.5%), indicating that perception is a major bottleneck for this architecture.
• Induction: In the symbolic setting, the VLM and Bayesian agents showed more modest gains, suggesting their primary failure modes lie in hypothesis formulation and revision rather than visual perception.
• Complexity: Performance gaps widened with rule complexity. VLM agents solved 67.5% of First-Order games but dropped to ~30% on Second-Order and Complex rules. Humans maintained a >50% win rate across all complexities.

3. VLM Agents Fail at Active Experimentation

VLM-based agents consistently proposed near-uninformative experiments, exhibiting near-zero Expected Information Gain (EIG) after roughly 10 examples. They tended to generate scenes similar to prior observations rather than actively probing the hypothesis space to reduce uncertainty. In contrast, the Oracle Agent maximized EIG by selecting experiments that best discriminated between competing hypotheses.

4. Human vs. AI Comparison

Humans outperformed all AI agents in end-to-end success (73.3% win rate vs. 44.5% for the best AI agent). While humans and VLM agents shared surface-level error patterns on simple rules, humans demonstrated superior stability in rule trajectories and successfully recovered complex and Out-of-Distribution (OOD) rules where no visual agent succeeded. Notably, 7 out of 10 human participants solved an OOD task (requiring logic outside the DSL), whereas no agent succeeded on visual inputs.

Significance and Claims

The paper claims that ZendoWorld represents a critical step toward evaluating intelligent agents by isolating the interplay between perception, induction, and active experimentation. The authors argue that current methods are highly specialized: some excel at labeling, others at structured search, but none reliably solve the full interactive loop without prior knowledge of the rule space.

The work identifies concrete avenues for improvement, particularly for domains like scientific discovery:

• Architectural Integration: The strict separation between perception and reasoning (as seen in the modular Oracle Agent) may be insufficient. The authors suggest that human-like conceptual thinking involves a mutual constitution of perception and reasoning, where hypotheses direct visual attention.
• Active Search: Future systems require architectures that combine visual grounding with structured, falsifiable hypotheses and active search strategies that prioritize information gain over mere consistency with past data.
• Evaluation Shift: The results suggest that evaluating inductive reasoning systems based solely on example labeling is insufficient; metrics must account for the ability to generate informative experiments and recover the underlying generative rules.

The authors conclude that bridging the gap between current AI capabilities and human-level active visual rule learning requires moving beyond passive reasoning and fixed perception modules toward systems capable of iterative hypothesis testing and dynamic feature attention.