Naturalistic Computational Cognitive Science: Towards generalizable models and theories that capture the full range of natural behavior

Technical Summary: Naturalistic Computational Cognitive Science

Problem Statement
Cognitive science faces a persistent challenge in building generalizable theories that span the full scope of natural situations and behaviors. While traditional approaches have sought generalizable models (e.g., connectionism, Bayesian inference, cognitive architectures), there is substantial debate regarding whether current theories are adequately constrained or capable of generalizing beyond simplified experimental paradigms. The authors argue that a reliance on simplified stimuli and tasks may lead to theories that fail to capture the computational mechanisms engaged by natural intelligence. Furthermore, the field lacks the infrastructure to systematically test models across a broad spectrum of naturalistic conditions, leading to fragmented findings that resist integration.

Methodology and Research Strategy
The paper proposes a research strategy termed "Naturalistic Computational Cognitive Science." This approach does not advocate for abandoning simplified experiments but rather for extending them by increasing ecological validity while maintaining experimental control. The methodology is structured around three core pillars:

1. Expanding Experimental Paradigms: The authors suggest broadening experimental parameters to better cover the scope of natural distributions. This involves:

   • Expanding existing parameters: e.g., increasing the category set in object classification to include daily life objects.
   • Adding ecologically motivated parameters: e.g., testing linguistic judgments across both spoken and written modalities, or introducing social agents and partially observable environments.
   • Parametric Manipulation of Naturalistic Data: Using generative AI and virtual environments (e.g., 2D Minecraft, VR) to create stimuli that preserve natural variation while allowing for controlled, parametric manipulation of specific features (e.g., decoupling shape and texture in natural images) to test hypotheses.

2. Developing Generalizable, Task-Performing Models: Drawing on Machine Learning (ML) engineering practices, the authors advocate for:

   • Dynamic Meta-Benchmarks: Moving beyond static datasets to "living collections" of benchmarks that researchers continually expand. These benchmarks should test specific facets of a cognitive capacity (e.g., shape vs. texture reliance) and evolve to include new hypotheses and naturalistic variations.
   • Iterative Development: Separating model design from task design to avoid overfitting. Models should be developed on simplified tasks but rigorously validated on increasingly naturalistic settings.
   • Frictionless Reproducibility: Adopting standardized evaluation protocols, single-line code data access (e.g., via Hugging Face), and "single-file" code philosophies to enable rapid, collective iteration and reuse of research artifacts.

3. Theory Construction via Reduction: The authors propose a dual-component theory structure:

   • Component 1: Task-performing (predictive) models that reproduce human behavior across a wide range of naturalistic stimuli and paradigms.
   • Component 2: Reductive explanations that link these complex models to simpler mechanisms, data properties, or formal theories (e.g., rational analysis, Bayesian models). This involves using mechanistic analysis (e.g., Distributed Alignment Search) and data property analysis to derive normative explanations for why the models behave as they do.

Key Findings and Evidence Reviewed
The paper synthesizes evidence from neuroscience, cognitive science, and AI to demonstrate the necessity of naturalistic approaches:

• Computational Mechanisms Shift: Naturalistic settings expose different computational challenges. For instance, credit assignment in large, delayed-consequence environments (like navigating a campus) requires mechanisms like "multitask preplay" that are not evident in small, grid-world tasks. Similarly, the role of the Medial Temporal Lobe (MTL) in oddity tasks depends on whether the visual cortex can resolve the stimulus, a distinction only clear when testing complex, naturalistic stimuli.
• Qualitative Behavioral Differences: Behavior changes qualitatively in naturalistic settings. Examples include:
  • Memory: Incidental visual search leads to better recall and spatial accuracy than explicit memorization.
  • Learning Efficiency: Mice learn 1000x faster in naturalistic labyrinth exploration compared to 2AFC tasks.
  • Moral Reasoning: Moral actions in Virtual Reality (VR) diverge significantly from moral judgments in text vignettes (e.g., the footbridge dilemma).
• Neural Coding Differences: Neural systems operate differently under naturalistic conditions. For example, the superior colliculus integrates multimodal cues for body control in head-free settings, whereas it is often studied only for saccadic control in head-fixed settings. Active viewing shifts the timing of visual responses (P100/M100) compared to passive viewing.
• Benefits of Naturalistic Data in AI: Training on naturalistic data yields qualitatively different generalization patterns.
  • Generalization: Agents interacting with 3D environments show better compositional generalization than image classifiers trained on screenshots.
  • Emergent Capabilities: Models trained on vast naturalistic data (e.g., Large Language Models, Reinforcement Learning agents) develop mechanisms for disparate tasks (e.g., arithmetic, video game playing) without explicit training signals.
  • Origins of Knowledge: Naturalistic data allows for probing the origins of cognitive knowledge, such as showing that face specialization can emerge from domain-general models trained on naturalistic distributions, challenging strict innateness hypotheses.

Key Contributions

1. Conceptual Framework: The paper defines "Naturalistic Computational Cognitive Science" as a strategy to build theories that predict behavior across increasingly naturalistic settings, moving beyond the limitations of simplified paradigms.
2. Engineering Roadmap: It translates ML engineering practices (dynamic meta-benchmarks, frictionless reproducibility, standardized interfaces) into actionable recommendations for cognitive science to foster collective, systematic progress.
3. Methodological Integration: It outlines how to combine controlled experimental manipulation (via parametric generation of naturalistic data) with complex, task-performing models to resolve theoretical conflicts (e.g., the MTL oddity debate).
4. Theoretical Synthesis: It proposes a specific architecture for theory-building that couples predictive task-performing models with reductive, mechanistic explanations, ensuring theories are both empirically grounded and theoretically interpretable.

Significance and Claims
The authors claim that increasing the naturalism of experimental settings and models is necessary, not merely beneficial, for developing a complete and generalizable theory of cognition. They argue that:

• Simplified settings often fail to capture the full range of computational challenges and mechanisms (e.g., credit assignment, long-horizon planning) required for natural intelligence.
• Current theories are often under-constrained because they are tested on narrow distributions; expanding the scope of testing via naturalistic paradigms and meta-benchmarks is essential to identify robust, generalizable principles.
• The integration of AI progress (in model engineering and data scale) with cognitive science (in theory and experimental design) offers a path to understanding the "roots" of cognitive and neural phenomena.
• By adopting these practices, the field can move toward theories that explain the full range of natural behavior while retaining the ability to provide reductive, mechanistic understanding.

The paper concludes that this perspective synthesizes growing literature on naturalistic experiments and generalizable models, offering a path to resolve long-standing debates about the generalizability of cognitive theories and the underdetermination of scientific models.