🤖 AI

Small Foundation Models of Human Cognition and Behaviour

This paper demonstrates that while small foundation models fine-tuned on human behavioral data can effectively match large-scale baselines for in-distribution psychological tasks, only larger models significantly improve generalization to novel task structures, with performance heavily reliant on processing specific stimulus and feedback content rather than just choice history.

Original authors: Nick Oh, Fernand Gobet

Published 2026-08-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nick Oh, Fernand Gobet

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to think like a human. For decades, scientists have built tiny, hand-crafted "mental machines" (like ACT-R or Soar) that try to explain exactly how our brains make decisions, step-by-step. But these machines are fragile; if you change the game slightly, the robot breaks. Recently, a new idea took over: instead of building a mental machine from scratch, why not just feed a giant, super-smart computer brain (a Large Language Model) millions of examples of real humans making choices? If the computer can predict what a human will do next, maybe it has learned the "rules" of human thinking. This is the world of "cognitive proxies"—AI models that act as stand-ins for human minds. But a big question remains: Does this giant brain actually understand the logic of the game, or is it just a master of pattern-matching, memorizing shortcuts like a student who only memorized the answer key without reading the textbook?

This paper, titled "Small Foundation Models of Human Cognition and Behaviour," dives into that mystery. The authors, Nick Oh and Fernand Gobet, decided to test if you really need a massive, 70-billion-parameter "giant" brain to mimic human behavior, or if a much smaller, "pocket-sized" model could do the trick. They trained fourteen different AI models, ranging from tiny (135 million parameters) to large (14 billion), on a massive dataset called Psych-101, which contains over 10 million choices made by people in 160 different psychological experiments.

Here is what they found, and it's a bit of a plot twist. First, they discovered that size doesn't matter as much as you'd think if you are just looking at the same types of games the AI has already seen. A tiny model with just 0.6 billion parameters could perform almost as well as the massive 70-billion-parameter giant when predicting choices in familiar experiments. It's like a small, nimble dog learning a specific trick just as fast as a huge Great Dane. However, the moment they threw a new type of game at them (out-of-distribution), the size suddenly mattered again. The bigger models were much better at figuring out the rules of a game they had never seen before, while the small ones got confused.

But the most exciting part is how these models are learning. Some critics worried these AIs were just relying on spurious correlations by ignoring the actual game rules and only looking at the history of past choices (like guessing the next card in a deck just by counting what's already been played). To test this, the authors played a game of "hide and seek" with the information. They systematically removed pieces of the prompt: they hid the instructions, they blurred the game stimuli (the pictures or numbers), they wiped out the feedback, and they even scrambled the order of the trials.

The results were dramatic. When they blurred the actual content of the game (the stimuli and feedback), the models' performance crashed, dropping below what you'd get by just guessing randomly. This proves the models aren't just memorizing a sequence of answers; they are actually paying attention to the content of the game. Furthermore, when they shuffled the order of trials in a game where order shouldn't matter, the models didn't care. But when they shuffled a game where order did matter, the models got confused. This suggests the AI is smart enough to know when the sequence of events is important and when it isn't.

In short, these small, fine-tuned models are excellent "noise ceiling" estimators. Think of a noise ceiling as a glass ceiling for how well we can predict human behavior. If a model hits this ceiling, it means we've found all the predictable patterns in the data, and any remaining errors are just random human "noise." These small models show that we can hit this ceiling without needing a supercomputer, but they also remind us that these models are only as good as the games they've been trained on. They are brilliant at mimicking human behavior in the lab, but they aren't yet a full theory of how the human mind works—they are a very accurate mirror, but not the person looking into it.

Technical Summary: Small Foundation Models of Human Cognition and Behaviour

Problem Statement
Large language models (LLMs) fine-tuned on human behavioural data, such as the 70-billion-parameter Centaur model, have emerged as general-purpose cognitive proxies. However, two critical questions remain unresolved regarding this paradigm:

Scale Necessity: Is the massive parameter count (e.g., 70B) required for predictive accuracy, or is it an artifact of specific training configurations?
Mechanism of Learning: Do these models genuinely process task structure and stimuli, or do they exploit statistical shortcuts (e.g., relying solely on choice history or prompt templates) to achieve high performance?

Previous work has suggested that models might rely on "shortcuts," yet no study has systematically decomposed the prompt to isolate the specific information channels used. Furthermore, the extent to which these models generalize to novel task structures (out-of-distribution) versus fitting the training distribution (in-distribution) is unclear.

Methodology
The authors trained fourteen models across four architecture families (Qwen3, Llama-3/3.2, SmolLM2/3, and OLMo2/3) with parameter counts ranging from 135M to 14B. These models were fine-tuned on Psych-101, a dataset comprising 10.7 million trial-level choices from over 60,000 participants across 160 psychological experiments.

The study employed a rigorous experimental design varying three factors independently:

Model Scale: Comparing models from 0.1B to 14B parameters.
Adapter Capacity: Sweeping LoRA ranks ( $r$ ) from 4 to 64 to determine if adapter capacity can substitute for model parameters.
Training Data: Using nested, experiment-stratified subsets of Psych-101 to isolate the effect of data quantity.

To diagnose what the models learn, the authors conducted structural ablation and permutation tests across 32 experiments:

Prompt Decomposition: The prompt was decomposed into four information channels: Task Instructions ( $I$ ), Experimental Stimuli ( $S$ ), Outcome Feedback ( $F$ ), and Choice History ( $C$ ).
Ablation Conditions: Models were tested under four conditions: Original, Instruction-ablated, Content-masked (replacing specific stimuli/feedback with generic placeholders), and History-only.
Order Permutation: Trial orders were permuted in two experiments—one with independent trials (exchangeable) and one with adaptive trials (non-exchangeable)—to test for sensitivity to sequential dependencies.

Key Results

In-Distribution Performance (Psych-101):
- Scale is not the limiting factor: Within the training distribution, model scale barely matters. Small models (0.6B to 1B parameters) with high adapter ranks (e.g., $r=32$ ) match or exceed the performance of a 70B baseline.
- Adapter Substitution: Adapter capacity substitutes for parameter count. A 0.6B model at rank 64 achieves the same performance as a 14B model at rank 4, despite having 23 times fewer parameters.
- Diminishing Returns: Increasing training data yields diminishing returns; the gain from doubling the dataset size decreases steadily.
Out-of-Distribution Performance (Psych-201-RT):
- Scale Reappears: When tested on unseen experiments, the performance gap widens significantly. Larger models demonstrate a clear advantage in generalizing to novel task structures.
- Transfer vs. Fit: Model size is a choice about transfer capability rather than in-distribution fit. The "noise ceiling" (the best achievable prediction) for in-distribution tasks is reached by small models, but this ceiling does not extend to out-of-distribution tasks without larger capacity.
Diagnostic Findings (Prompt Decomposition):
- Content is Critical: Masking the content of stimuli and feedback destroys 75.7% of the learned information, pushing models below chance performance. This refutes the "history-only" shortcut hypothesis.
- Template vs. Content: Removing instructions costs little (12.5% loss), but removing the content of the template costs almost everything. The models rely on the specific information provided to the participant, not just the structural format.
- Task-Adaptive Sensitivity: Models exhibit order invariance on tasks with independent trials (exchangeable) but remain sensitive to trial order in adaptive tasks. This indicates the models learn the computational structure of the specific tasks rather than applying a uniform sequential heuristic.

Significance and Claims
The paper posits a modest but specific role for these small, cognitively fine-tuned models:

Noise Ceiling Estimators: These models serve as empirical bounds (noise ceilings) for psychological experiments. They estimate how much of the behavior in a given paradigm is predictable, thereby identifying the "gap" where current interpretable theories fall short. This aligns with the concept of "scientific regret minimization."
Not Cognitive Theories: The authors explicitly state these models are not theories of cognition in the tradition of ACT-R or Soar. They posit no architecture, commit to no mechanism, and do not explain how a choice is generated.
Validity of Instrument: The models are valid instruments because their predictions rest on the actual information participants used (stimuli and feedback), as proven by the ablation studies. However, their utility is bounded by the paradigms seen in training; they do not generalize reliably beyond the scope of Psych-101 without significant scaling.

In conclusion, the study demonstrates that 70 billion parameters are not necessary for fitting human behavioral data within known paradigms, but larger scales are required for transfer to novel structures. The resulting small models offer a data-driven reference point for evaluating and improving process-level cognitive theories, provided their scope is understood as an estimator of predictability rather than a mechanistic explanation.

More like this