Comparing Semantic Navigation in Humans and Large Language Models using Natural Language Processing

Technical Summary: Comparing Semantic Navigation in Humans and Large Language Models

Problem Statement
Semantic memory retrieval is conceptualized as a navigational process through conceptual space. While previous research has utilized the Verbal Fluency Task (VFT) to model human semantic search—often framing it as a balance between local exploitation (clustering) and global exploration (switching)—recent advances in Large Language Models (LLMs) offer a new avenue for comparison. Although LLMs generate coherent language and their internal representations show some alignment with human brain activity, it remains unclear whether their underlying search dynamics mirror human cognitive processes. Prior work indicates that LLM-derived semantic networks often exhibit weaker local clustering and poorer global integration than human networks. However, a gap exists in quantifying how these systems progress through semantic content on a step-by-step basis, particularly using metrics that account for the accumulating context of retrieval history rather than treating words as independent static points.

Methodology
The study employed a comparative design using behavioral data from 82 human participants and outputs from three state-of-the-art LLMs: GPT-4o, Gemini-2.5-Pro, and Claude-Sonnet-4.5.

• Task: All agents performed a semantic verbal fluency task for the category "animals." Human data was sourced from the SNAFU database (3-minute limit). LLMs were prompted using a role-playing induction (adopting specific occupational personas) to increase output diversity. To ensure comparability without imposing artificial time limits on models, LLM output lengths were matched to the empirical distribution of human response counts.
• Experimental Manipulation: LLMs were tested across eight temperature settings (0.0 to 1.0) to modulate sampling stochasticity, ranging from deterministic to highly diverse outputs.
• NLP Metrics: The study applied a physics-inspired, trajectory-based framework (Toro-Hernández et al., 2026) using cumulative contextual embeddings (via OpenAI's text-embedding-3-large, validated with other systems). Three complementary metrics were calculated:
  1. Distance to Next: The cosine distance between successive cumulative semantic states, measuring the magnitude of "semantic jumps" between consecutive items.
  2. Entropy: The Shannon entropy of the step-size distribution, quantifying the predictability and variability of the search trajectory.
  3. Distance to Centroid: The average cosine distance of non-cumulative item embeddings from the centroid of the unique items generated, indexing global dispersion or clustering.

Key Results

• Human vs. LLM Baseline: Humans exhibited significantly higher entropy (less predictable step sizes), larger semantic steps (Distance to Next), and broader global dispersion (Distance to Centroid) compared to all three LLMs. This indicates that human search is more variable and exploratory, whereas LLMs demonstrate more rigid, locally clustered, and conservative navigation.
• Model Differences: Among LLMs, Claude-Sonnet-4.5 showed the smallest step sizes and tightest local clustering. GPT-4o exhibited the strongest global clustering (lowest Distance to Centroid).
• Temperature Effects: Tuning temperature produced only partial alignment with human profiles.
  • Entropy and Distance to Next: Alignment was model-specific. Claude approached human levels at high-intermediate temperatures (0.75–0.90), while Gemini and GPT-4o showed convergence at endpoints (0.0, 1.0) and specific intermediate points.
  • Distance to Centroid: This metric was the most difficult to match. Only narrow temperature bands for Claude and Gemini were statistically indistinguishable from humans. GPT-4o remained persistently non-human-like across all settings, showing consistently reduced global dispersion.
• Integrated Profile: No single temperature configuration allowed any LLM to reproduce the complete human profile across all three dimensions simultaneously.

Significance and Claims
The paper argues that human semantic search implements a distinctive balance between local exploitation and global exploration that current LLM architectures fail to reproduce, even with parameter tuning. The findings challenge the notion of LLMs as straightforward "silicon participants" in cognitive research. The authors suggest that the systematic deviations observed—specifically the inability to simultaneously match unpredictability, step magnitude, and global dispersion—likely stem from the training objectives of modern LLMs (e.g., Reinforcement Learning from Human Feedback), which optimize for assistant-like performance rather than faithful replication of human cognitive processes.

The study concludes that while LLMs can mimic human behavior on isolated metrics under specific conditions, they lack the integrated behavioral signature of human semantic navigation. Consequently, researchers should exercise caution when using LLMs as cognitive surrogates without extensive validation across multiple theoretically motivated dimensions, as calibrating for one metric may distort others. The results highlight the need for future work to explore whether alternative training objectives or architectural modifications could better capture the computational principles underlying human semantic memory search.