Closing the Loop to Discover Psychological Theories with an Automated Cognitive Scientist

This paper introduces AutoCog, a fully autonomous AI system that closes the loop of scientific discovery in cognitive science by automatically generating, testing, and refining executable cognitive theories through iterative cycles of experiment design, data collection, and model synthesis, ultimately producing novel, empirically validated insights into decision-making.

The Gist

Imagine a team of scientists trying to figure out how the human mind makes decisions. Usually, this is a slow, manual process: a human researcher guesses a theory, designs a test, runs it on people, sees if they were right, and then tries to write a better theory based on the mistakes.

This paper introduces AUTOCOG, a fully automated "robot scientist" that does all of this by itself, closing the loop from start to finish without human help. Think of it not as a single robot, but as a self-driving laboratory for the mind.

Here is how it works, using simple analogies:

1. The Setup: Two Rival Chefs

Imagine you want to find the perfect recipe for a cake, but you don't know the ingredients. You start with two rival chefs (the "seed theories").

• Chef A thinks the secret is just sugar.
• Chef B thinks the secret is just flour.

In the past, a human would have to bake cakes, taste them, and tell the chefs how to improve. With AUTOCOG, the system acts as a master critic and a relentless editor. It takes these two chefs and says, "Okay, let's see who is right."

2. The Cycle: The Robot's Daily Routine

AUTOCOG runs in a continuous loop of four steps, like a video game level that repeats:

• Step 1: Designing the Ultimate Test (The Trap)
  The system acts like a game designer. It looks at what Chef A and Chef B believe and designs a specific "taste test" where they will definitely disagree. If Chef A says "Add more sugar" and Chef B says "Add more flour," the robot creates a cake scenario where only one of them can be right. It's like setting up a maze where only one path leads to the exit.

• Step 2: Gathering the Evidence (The Taste Test)
  The robot runs this test. It can do this in two ways:

  • Simulation: It asks a computer model (a "digital human") to play the game.
  • Real Humans: It recruits real people online to make choices (like picking between two products with different expert ratings).
    The robot collects the data: "Who won? What did they choose?"

• Step 3: The Verdict (The Critic)
  The robot analyzes the results. It asks: "Did Chef A's recipe match the taste? Did Chef B's?"
  If the data falls somewhere in the middle, the robot doesn't just pick a winner. It acts like a detective, asking, "Why did both chefs fail? What are they missing?" It writes a report explaining exactly where the theories broke down.

• Step 4: The Rewrite (The Evolution)
  Based on the verdict, the robot rewrites the losing chef's recipe. It doesn't just tweak a little bit; it might invent a completely new cooking method. It creates a "successor" theory that tries to fix the mistakes. Then, the loop starts again with this new, improved chef.

3. The Results: What Did the Robot Discover?

The researchers tested this robot in the world of decision-making (how people choose between options).

• The Memory Test: First, they fed the robot fake data generated by known "perfect" strategies. The robot successfully figured out the hidden rules, even when the data was noisy (like people making random mistakes). It proved the robot could "learn" the truth from the data, not just guess based on what it was told.
• The Real Discovery: When they let the robot test on real humans, something amazing happened. The robot didn't just find the theories humans already knew (like "Take the Best" or "Weighted Average"). It found something new.

The New Theory:
The robot discovered that when people look at ratings (like 1 to 5 stars), they don't treat every point equally.

• The Old View: A jump from 4 to 5 stars is the same "amount" of improvement as a jump from 1 to 2 stars.
• The Robot's Discovery: People feel a jump from 1 to 2 stars is huge, but a jump from 4 to 5 stars feels small. This is called "diminishing sensitivity."

To prove this wasn't a fluke, the researchers took the robot's new theory, wrote it down, and ran a pre-registered study (a strict, planned experiment) with new people. The people behaved exactly as the robot predicted. The robot had found a new law of human psychology.

4. Why This Matters

The paper claims this is the first time a system has fully automated the creative part of science.

• Usually, AI helps analyze data or run experiments, but a human has to come up with the idea of the theory.
• AUTOCOG comes up with the idea, tests it, realizes it's wrong, and invents a better one, all on its own.

It turns theory-building from a manual art into an executable, cumulative science. The robot keeps a log of every step, so humans can look back and see exactly how the robot "thought" its way to a new discovery.

In short: AUTOCOG is a robot that argues with itself, tests its own ideas on real people, learns from its failures, and eventually discovers new rules about how the human mind works—rules that humans hadn't thought of yet.

Technical Summary

Technical Summary: Closing the Loop to Discover Psychological Theories with an Automated Cognitive Scientist

Problem Statement

Scientific discovery in psychology has historically been bottlenecked by the creative, manual process of theory-building. While data collection, modeling, and experimental design have seen increasing automation, the core step of generating new theories—turning the accumulated failures of existing models into better ones—remains a human-centric art. Existing automated discovery systems in other fields (e.g., chemistry, materials science) often automate single stages or search fixed model spaces over static datasets. In cognitive science, the challenge is conceptual: theories of the mind cannot be compiled or solved directly but must navigate an open conceptual space to render opaque psychological processes mechanistically meaningful. The central problem addressed is the lack of a fully autonomous system capable of closing the discovery loop for human behavior, specifically the ability to propose competing theories, design discriminating experiments, collect data, and synthesize improved theories without human intervention.

Methodology: The AUTOCOG System

The authors present AUTOCOG (AUTOmated COGnitive scientist), a fully autonomous agentic-AI system designed to close the discovery loop. The system operates as a closed-loop pipeline iterated over $N$ cycles, consisting of four key stages driven by Large Language Model (LLM) agents:

1. Initialization: A researcher specifies the task domain, instructions, experiment design space, model design space, seed theories, and an optional metric.
2. Experimental Design: An LLM agent proposes an experiment where two incumbent theories make divergent predictions, adhering to the principle of adversarial experimental design. A second agent proposes a metric to quantify the predicted advantage of one theory over the other. The system validates these proposals via forward simulation before data collection to ensure discriminability.
3. Behavioral Data Collection: The committed experiment is executed on a data source. In this study, data was collected from three sources:
   • Simulated behavior from known cognitive models (ground truth recovery).
   • Foundation models (as in-silico stand-ins).
   • Human participants recruited online (via Prolific).
4. Analysis & Arbitration: The system compares the generative behavior of incumbent theories against observed data using pre-registered metrics. An LLM-based arbitration agent interprets the findings, diagnosing why a theory failed and recommending revisions.
5. Theory Revision: Based on the arbitration verdict, a separate LLM agent revises the weaker theory. This involves either re-synthesizing the associated cognitive model or regenerating the theory from scratch using program synthesis (producing executable Python code). The revised theory re-enters the loop.

Key Design Principles:

• Open-ended Search: The system searches the space of theories and experiments rather than selecting from a fixed candidate set.
• Generative Comparison: Theories are compared based on their ability to generate behavior (forward simulation) rather than fitting parameters to observed data, reducing overfitting risks.
• Self-Verification: Every proposal (experiment, theory, metric) is verified against its own predictions via simulation before observing actual data.
• Generalization Pressure: Theories must capture behavior across multiple populations and tasks to survive, acting as an implicit pressure for parsimony and unification.

Key Contributions

1. First Autonomous Closed-Loop System for Cognitive Theory: AUTOCOG is presented as the first system to autonomously discover theories of human cognition directly from human behavior, closing the loop from experiment design to theory revision without researcher intervention.
2. Executable Theory Representation: The system treats theories as natural-language mechanistic claims paired with executable Python programs, allowing for rigorous simulation and verification.
3. Adversarial Discovery Framework: The system utilizes LLM agents to advocate for competing theories and design experiments specifically to discriminate between them, mimicking the adversarial nature of scientific debate.

Results

1. Recovery of Known Decision-Making Strategies

In simulations where ground-truth strategies were known, AUTOCOG successfully recovered the generating theories even when seeded with distinct, competing theories.

• Canonical Strategies: The system recovered Take-the-Best (TTB), Tallying, and Weighted-Additive (WADD) with high fidelity (Mean Squared Error indistinguishable from ground truth in noise-free conditions).
• Non-Canonical Strategies: The system successfully recovered unconventional strategies (e.g., "Take-the-worst," "Alternating," "Perseveration") within 5–20 cycles, demonstrating it is not strictly bound by the priors of the underlying LLMs.
• Noise Robustness: Performance degraded gracefully with increasing action noise ($\epsilon$), and the system converged on random-choice models when presented with pure noise, indicating it does not impose structure where none exists.

2. Discovery in Multi-Attribute Decision-Making (Binary Cues)

When run with human participants on a binary-valued cue task (seeded with TTB and WADD), AUTOCOG surfaced a Non-linear Subjective Weighting Model.

• Performance: This surfaced theory significantly outperformed the seed theories (TTB, Tallying, WADD) on held-out datasets from the literature (Hilbig et al., 2014).
• Mechanism: The model unified the three canonical heuristics as limiting cases of a single mechanism governed by a power-law transformation of cue validities (an exponent parameter). It achieved this without explicit parameter optimization or parsimony constraints, driven solely by the need to generalize across accumulated data.

3. Discovery in Multi-Attribute Decision-Making (Cardinal Cues)

Expanding the design space to include cardinal-valued cues (integers 0–$r_{max}$), AUTOCOG discovered a Diminishing Returns WADD model.

• Novelty: This theory posits that individuals apply a concave utility function to cue values before weighting them by validity. This implies diminishing sensitivity to feature values: a unit difference in low-value regions has a larger impact on choice than the same difference in high-value regions.
• Validation: The system's discovery was prospectively tested in a preregistered study with new human participants. The study confirmed three hypotheses derived from the theory:
  • H1 (Model Discrimination): Participants chose options predicted by Diminishing Returns WADD significantly more often than those predicted by linear WADD, Tallying, or TTB.
  • H2 (Steep-vs-Flat): Participants showed greater sensitivity to advantages in the "steep" (low-value) region of the scale compared to the "flat" (high-value) region.
  • H3 (Level-Shift): The same objective cue-value structure exerted a stronger influence when presented in a low-value range compared to a shifted-up high-value range.

Significance and Claims

The paper claims that AUTOCOG demonstrates how automated discovery systems can transform cognitive theory-building into an explicit, executable, and cumulative science.

• Automation of the Creative Step: The system automates the "creative" step of theory generation, a task previously considered uniquely human.
• Data-Driven Discovery: The discoveries (including the novel diminishing sensitivity finding) were driven by the collected behavioral data rather than the priors of the LLMs, as evidenced by the system's ability to recover non-canonical strategies and its failure to find structure in pure noise.
• Auditability: The entire discovery trajectory is logged as a machine-readable trace, making the path of scientific reasoning open to inspection and allowing other agents or researchers to resume the search from the frontier.
• Scope: The authors note that the system is bounded by the experiment and model design spaces specified by humans and that the translation from verbal theory to executable code remains a challenge. However, they assert that discovering psychological theories of human behavior is now within the reach of automation.

The work does not claim to replace human scientists but rather to shift their role from execution to specification—deciding what counts as a good theory and defining the search space—while the system handles the iterative generation and testing of candidates.