Technical Summary: auto-psych

Problem Statement

The scientific process in cognitive science is traditionally an iterative loop involving theory conjecture, experimental design, data collection, and model refinement. While AI tools have recently assisted in individual steps of this loop (e.g., generating hypotheses or analyzing existing datasets), a critical bottleneck remains: the automation of new data collection under targeted experimental regimes. Current AI-driven science often relies on "Box's loop" (iterative model testing on static datasets) but lacks a "lab-in-the-loop" capability where agents design and execute experiments with human participants.

This paper addresses the challenge of automating the complete discovery loop in computational cognitive science. Specifically, it targets the domain of subjective randomness—the psychological phenomenon where humans judge which sequences of coin flips appear more random. This domain is chosen because it offers a discrete, tractable stimulus space and a rich history of formalized probabilistic theories, making it an ideal testbed for automated theory discovery.

Methodology: The auto-psych Framework

The authors introduce auto-psych, an agent-based system that automates theory discovery and experimentation through two nested loops. The system utilizes Large Language Models (LLMs), specifically gemini-3.1-pro-preview within an OpenCode harness, to instantiate distinct agent roles (Theorist, Critic, Design, and Implementation agents).

1. The Outer Loop: "Lab-in-the-Loop"

The outer loop manages the experimental cycle to differentiate between competing theories:

• Theory Generation: A theorist agent proposes new cognitive models. In the first iteration, it uses "seed models" from the literature. In subsequent iterations, it generates new hypotheses in natural language and translates them into executable PyMC (probabilistic programming) code.
• Experiment Design: A design agent generates 100–300 candidate stimuli (pairs of coin flip sequences). These are scored using Expected Information Gain (EIG) to select the pairs that most effectively distinguish between the current set of models in the registry.
• Data Collection: An implementation agent constructs a jsPsych experiment, deploys it to Firebase, and recruits human participants via the Prolific API. The system monitors the study, retrieves data, and passes it to the inner loop.

2. The Inner Loop: "Box's Loop"

The inner loop refines the best-performing model from the outer loop before it is added back to the registry:

• Model Fitting: Existing models are fit to the collected human data using Markov Chain Monte Carlo (MCMC).
• Critique: A critic agent (adapted from the CriticAL framework) performs posterior-predictive checks. It simulates data from the best-fitting model and computes test statistics to identify significant discrepancies between the model's predictions and the observed human data.
• Refinement: A theorist agent uses these specific critiques to propose new models or refine existing mechanisms to address the identified failures. This loop runs twice per outer iteration.

Key Contributions

1. Full Automation of the Discovery Loop: The paper presents the first instance of a fully automated iterative psychology loop where AI agents design experiments, recruit human participants, collect real behavioral data, and refine computational theories without human intervention in the experimental cycle.
2. Nested Agent Architecture: The system demonstrates that a nested structure (an inner loop for model critique/refinement and an outer loop for experimental design) is critical for performance. Ablation studies show that removing the inner loop significantly degrades the system's ability to recover ground-truth models.
3. Integration of Probabilistic Programming: The framework successfully bridges LLMs with probabilistic programming languages (PyMC), allowing agents to generate, fit, and critique complex cognitive models that represent human cognition as probabilistic processes.

Results

The system was evaluated in two primary ways:

1. Ground-Truth Recovery

The authors tested auto-psych's ability to recover known models (both literature-based seed models and "alien" models with psychologically implausible rules) from synthetic data generated by those models.

• Performance: The system consistently recovered the ground-truth models with higher accuracy (lower Root Mean Squared Error) than the best-fitting seed models.
• Necessity of Nested Loops: Ablation experiments confirmed that the inner loop is essential; without it, the system's recovery performance dropped significantly.

2. Human Data Experiments

The system ran three independent replicates of the discovery loop with human participants (40 participants per round, 3 rounds per replicate).

• Superiority over Literature Models: In all three replicates, the agents discovered models that fit the human data significantly better than the initial seed models derived from the scientific literature (e.g., Bayesian diagnosticity, prototype similarity, encoding compressibility).
• Novel Discoveries: The best-performing discovered model was the "Minkowski typicality" model. This model posits that humans judge randomness by comparing a sequence's alternation rate and head proportion to a mental prototype, with deviations weighted by a penalty power exponent (posterior mean ~1.35, suggesting a distance metric between Manhattan and Euclidean).
• Generalization: The discovered models maintained their superior fit when tested on held-out data from other replicates, indicating they were not overfit to the specific datasets used during their discovery.
• Diversity of Solutions: While different replicates converged on models with similar predictive power (low pairwise RMSE between best models), the internal structures varied (e.g., some based on prototype comparison, others on evidence accumulation), highlighting the non-unique nature of model discovery in high-dimensional spaces.

Significance and Claims

The paper claims to demonstrate the feasibility of automated data collection and theory discovery in computational cognitive science. It argues that:

• Efficiency: Automated systems can rapidly iterate through hypothesis generation and experimental testing, potentially accelerating the pace of scientific progress.
• Discovery Potential: The system can find theories that outperform established human-generated models, suggesting that AI agents can identify novel mechanistic explanations for human behavior that may have been overlooked.
• Scalability: While the current study is a proof-of-concept on a specific, restricted domain (subjective randomness), the framework is designed to be generalizable to other paradigms where theories can be formalized in code and experiments can be run online.

The authors remain modest regarding the scope, acknowledging that the current system operates within a restricted stimulus space and that the discovered models, while statistically superior, may still lack the deep explanatory virtues preferred by human psychologists. They frame this work as a starting point for integrating automated discovery systems into the broader scientific infrastructure rather than a replacement for human scientists.