Can Large Language Models Simulate Human Cognition Beyond Behavioral Imitation?

1. Problem Statement

The central question addressed by this research is whether Large Language Models (LLMs) can genuinely simulate human cognition (internalizing reasoning patterns, problem-solving strategies, and cognitive biases) or if they merely imitate surface-level behaviors (style, vocabulary, and specific methods).

Existing benchmarks fail to answer this because:

• Synthetic Data: Many role-playing datasets use LLMs to generate reasoning traces, creating a circular dependency that does not reflect authentic human cognition.
• Aggregation: Human-annotated benchmarks often mix data from different individuals, collapsing unique cognitive patterns into population-level statistics, thereby destroying individual cognitive consistency.
• Data Scarcity: Authentic human cognitive data (step-by-step internal reasoning) is difficult to collect in natural settings.

2. Methodology

A. The Cognitive Benchmark Framework

The authors constructed a novel benchmark grounded in the longitudinal research trajectories of 217 prominent AI researchers.

• Data Source: Scientific publications (specifically Introduction sections) serve as "externalized representations" of cognitive processes. The authors identified four cognitive elements in these texts: Background, Motivation, Approach, and Solution.
• Cognitive Subject: Each researcher is treated as a distinct cognitive subject.
• Cross-Domain Generalization Setting:
  • Train Set: Historical research topics of an author.
  • Test Set: The author's most recent, previously unseen research topic.
  • Goal: Force the LLM to solve a new problem in a novel domain while maintaining the cognitive consistency of the specific author, rather than just mimicking their writing style or past specific methods.

B. Evaluation Framework

To distinguish between behavioral imitation and cognitive internalization, the authors proposed a Multi-Level Evaluation Framework:

1. Lexical Coverage: ROUGE and BertScore (measures surface similarity).
2. Semantic Similarity: LLM-as-a-Judge (measures alignment of ideas).
3. Cognitive Consistency (The Core Metric): A novel set of five dimensions based on psychological theories (Witkin's field dependence/independence and Kirton's adaptation-innovation):
   • Foundational Premise: Consistency in problem framing and theoretical lenses.
   • Strategic Preference: Decision-making tendencies in technical trade-offs (e.g., theory vs. engineering).
   • Logic Mapping: Consistency in reasoning structures (e.g., deductive vs. empirical).
   • Keyword Style: Specific lexical preferences and terminology habits.
   • Taboo Avoidance: Adherence to methods the author historically critiques or avoids.

C. Experimental Setup

• Models: Evaluated state-of-the-art LLMs (LLaMA-3.1, Qwen-3, GPT-5.1, Gemini-2.5, etc.).
• Conditions:
  • BASE: No history.
  • OTHER: Random author history (control).
  • SELF: Target author's historical history (cross-domain).
  • SELFAKIN: Target author's recent in-domain history (behavioral leakage check).
  • LEAKY: Oracle access to the ground-truth cognitive trajectory (upper bound).
• Enhancement Techniques: Tested prompting strategies (Personality Traits, Cognitive Guidance) and learning-based methods (Individual fine-tuning, Unified profile-conditioned training).

3. Key Results

A. Behavioral Simulation vs. Cognitive Internalization

• Behavioral Simulation (Yes): LLMs can successfully mimic an author's writing style and specific methods when provided with relevant history (SELF and SELFAKIN settings show improvements over OTHER and BASE in lexical and semantic scores).
• Cognitive Internalization (No): LLMs fail to internalize and transfer abstract cognitive patterns across domains.
  • The performance gap between SELF (cross-domain) and SELFAKIN (in-domain) is significant. LLMs struggle to generalize the author's reasoning style to new topics.
  • Even with LEAKY (oracle) access, models do not reach the level of authentic human reasoning, suggesting they rely on statistical pattern matching rather than structural cognitive modeling.
  • Human evaluation confirms that ~95% of LLM outputs are rated as "behavioral imitation" or "failed cognitive simulation," with successful internalization being exceedingly rare.

B. Effectiveness of Enhancement Techniques

• Prompting (Limited): Techniques like PersonalityTrait and CognitiveGuide provide moderate gains in cognitive consistency but often degrade lexical metrics. They reshape expression but do not induce deep cognitive abstraction.
• Learning-Based Methods (Ineffective):
  • TrainIndividual: Captures surface-level statistical regularities rather than abstract patterns.
  • TrainUnified: Shows substantial lexical improvement but causes a systematic collapse in cognitive-level performance. Averaging individual traits in massive training erases specific cognitive signatures.

4. Key Contributions

1. New Benchmark: A large-scale, rigorous benchmark using authentic researcher publication histories to evaluate individual-level cognitive alignment, avoiding synthetic data and population-level aggregation.
2. Cognitive Metrics: Introduction of a multi-dimensional "Cognitive Consistency" metric that evaluates deep reasoning patterns (premises, strategies, logic) rather than just text similarity.
3. Empirical Evidence: A systematic study demonstrating that current SOTA LLMs are excellent at behavioral mimicry but fundamentally limited in cognitive internalization.
4. Paradigm Critique: The findings suggest that current "passive scaling" and "outcome-based" training paradigms are insufficient for genuine cognitive modeling.

5. Significance and Conclusion

The paper concludes that while LLMs have achieved remarkable behavioral alignment, they have not yet crossed the threshold into simulating human cognition. The current reliance on passive data exposure and statistical fitting allows models to approximate human outputs without replicating the underlying cognitive structures.

Implications for Future Research:

• Moving beyond passive scaling toward active learning objectives.
• Developing cognitive meta-learning and abstraction-oriented self-supervision.
• Shifting from outcome-based rewards to process-aware objectives that incentivize cognitively plausible intermediate reasoning steps.
• The authors suggest that achieving true human-level cognitive simulation may require brain-inspired architectures that go beyond current probabilistic modeling paradigms.

This work serves as a critical reality check for the AI community, clarifying the boundary between "acting like a human" and "thinking like a human."