Think-Aloud Reshapes Automated Cognitive Model Discovery Beyond Behavior

This paper demonstrates that incorporating Think-Aloud traces into automated cognitive model discovery significantly improves predictive performance and shifts the identified model structures toward more integrated utility mechanisms, revealing cognitive processes that behavioral data alone cannot recover.

The Gist

Imagine you are trying to figure out how a friend decides what to eat for dinner. You have two ways to learn about their process:

1. The "What" (Behavior): You watch them order. They pick the pizza. You see the result.
2. The "How" (Think-Aloud): You ask them to talk through their thoughts while deciding. They say, "Hmm, I'm hungry, but pizza is heavy. Maybe I should check the calories first, then compare the cost."

For a long time, scientists trying to build computer models of human thinking have only had access to the "What." They watch people make choices (like picking a risky gamble or a safe one) and try to reverse-engineer the math behind it.

The problem is that the "What" is often a foggy mirror. Many different internal math formulas can produce the exact same final choice. It's like seeing a car drive down a street; you know it moved from A to B, but you don't know if the driver was using a GPS, a map, or just guessing. This makes the computer models "under-determined"—there are too many possible answers, and the computer might pick the wrong one just because it fits the data okay.

The New Approach: Listening to the Inner Monologue

This paper introduces a new way to build these models. Instead of just watching the final choice, the researchers fed the computer models the "How" as well—the actual spoken thoughts (Think-Aloud traces) people had while making decisions.

They used a super-smart AI (a Large Language Model) to act as a detective. The AI was given two types of clues:

• Clue A: The list of choices the person made.
• Clue B: The transcript of what the person said while making those choices.

The AI then tried to write a computer program that could explain both the choices and the spoken thoughts.

What They Found

The researchers tested this on people making risky decisions (like choosing between a sure small reward or a chance at a big reward). Here is what happened when they added the "spoken thoughts" to the mix:

1. The Models Got Smarter (Better Predictions)
When the AI used only the choices, it made decent guesses. But when it used the choices plus the spoken thoughts, the models became much better at predicting what the person would do next time. It's like a detective solving a crime: if you only see the footprints, you might guess the wrong suspect. But if you also hear the suspect's alibi, you can pinpoint the truth much more accurately.

2. The Models Changed Their "DNA" (Structural Shift)
This is the most surprising part. The AI didn't just tweak the numbers; it completely changed the type of logic it used to explain the human mind.

• Without the spoken thoughts: The AI mostly thought humans were using a "Tug-of-War" method. It assumed people calculated the value of Option A, calculated the value of Option B, and then simply compared the two numbers to see which was bigger.
• With the spoken thoughts: The AI realized that for most people (about 70%), the brain works more like a "Smoothie Blender." Instead of just comparing two separate numbers, people were actually mixing the ingredients (risk, reward, probability) inside each option first, blending them into a single feeling, and then making a choice.

The paper found that for nearly 7 out of 10 people, adding the spoken thoughts forced the AI to abandon the "Tug-of-War" model and switch to the "Blender" model.

The Big Takeaway

The main point of this paper is that listening to how people think changes the map we draw of their minds.

If you only look at the destination (the choice), you might draw a map that looks like a straight line. But if you listen to the traveler's commentary, you realize they took a winding path, stopped to look at a view, and maybe even backtracked.

By adding "Think-Aloud" data, the researchers didn't just get a slightly better map; they discovered that the terrain itself was different than they thought. The spoken words acted as a constraint, forcing the computer to stop guessing and start finding the actual mental machinery people were using—machinery that was invisible if you only watched their hands.

Technical Summary

Technical Summary: Think-Aloud Reshapes Automated Cognitive Model Discovery

Problem Statement
Computational cognitive models derived solely from behavioral data are often under-determined; distinct computational mechanisms can produce identical or highly similar patterns of choice, leading to substantial ambiguity in model selection (Wilson & Collins, 2019). While "think-aloud" protocols have long been used to capture process-level reasoning in natural language, previous work has largely focused on validating the reliability of these verbal reports or developing methods to analyze them. A fundamental question remains unexplored: Can think-aloud reasoning traces facilitate the automated discovery of computational structures that are irrecoverable from behavioral data alone?

Methodology
The authors employ an automated model discovery framework called GeCCo (Rmus et al., 2025), which utilizes a Large Language Model (LLaMA-3.1-70B) to iteratively generate candidate computational models. These models are defined as executable functions mapping task inputs to choices. The LLM is prompted to propose alternative model structures that improve upon the current best-performing model based on their fit to held-out data.

The study applies this framework to a risky decision-making dataset involving 72 participants. In this task, participants make binary choices across 19 trials based on the paradigm established by Kahneman and Tversky (1979). The experiment compares two conditions:

1. Behavior-Only: Models are discovered using only the behavioral choice data.
2. Behavior + Think-Aloud: Models are discovered using both behavioral data and the participants' verbalized reasoning traces as input to the LLM.

For each participant, the discovery process is repeated 10 times with identical data splits. The best-fitting model for each condition is selected based on the Bayesian Information Criterion (BIC) (Watanabe, 2013) evaluated on 10 held-out trials. To analyze model structure, the discovered programs are converted into normalized computation graphs, from which structural features are extracted and clustered using HDBSCAN (McInnes et al., 2017).

Key Results

1. Improved Predictive Performance: Models discovered with the inclusion of think-aloud data demonstrated significantly better out-of-sample predictive performance. A paired t-test revealed that the held-out BIC scores were significantly lower for the think-aloud condition compared to the behavior-only condition ($t(71) = -3.41, p = 0.001$). Specifically, 59.7% of participants showed lower BIC under the think-aloud condition.
2. Systematic Reshaping of Model Structure: The inclusion of process-level language data did not merely refine existing models but systematically altered the structural class of the discovered mechanisms. Clustering identified three primary mechanism families: Integrated Utility, Explicit Comparator, and Rule-based Operator.
   • Structural Shifts: 69.4% of participants were assigned to different mechanism clusters when moving from the behavior-only condition to the think-aloud condition.
   • Specific Transitions: The most prominent shift occurred from the Explicit Comparator cluster (which computes utilities and compares them directly, e.g., $\Delta U = U_A - U_B$) to the Integrated Utility cluster (which transforms and integrates options before comparison).
   • Qualitative Differences: These transitions represent fundamental changes in computational organization (e.g., shifting from direct value comparison to a process of transforming and integrating gains/losses within each option) rather than superficial code variations.

Significance and Claims
The paper claims that process-level language data serves a functional role in constraining the space of admissible computational models. By providing additional constraints on the underlying cognitive mechanisms, think-aloud traces resolve the indeterminacy inherent in behavioral data alone.

The authors conclude that incorporating think-aloud data:

• Improves the predictive quality of automated model discovery.
• Systematically reshapes the structure of discovered cognitive models, leading to the identification of mechanism families that are not recoverable from behavior alone.
• Suggests that verbal reports are not just supplementary data but are critical for uncovering the true computational architecture of human decision-making.

The work demonstrates that automated model discovery frameworks can leverage natural language traces to move beyond behavioral under-determination, revealing distinct cognitive mechanisms that would otherwise remain hidden.