From Associations to Activations: Comparing Behavioral and Hidden-State Semantic Geometry in LLMs

1. Problem Statement

The paper addresses a fundamental question in cognitive science and AI interpretability: To what extent can a Large Language Model's (LLM) internal semantic geometry (hidden states) be recovered solely from its observable behavior?

While cognitive science traditionally treats semantic knowledge as a latent structure inferred only through behavior (e.g., word associations), LLMs offer a unique "fully observable" setting where both behavior and internal representations are accessible. The authors investigate whether discrete behavioral outputs (without access to logits or internal weights) contain sufficient information to reconstruct the model's internal semantic structure. Specifically, they compare behavioral semantic geometry (derived from association tasks) against layer-wise hidden-state geometry across multiple models.

2. Methodology

Experimental Setup

• Models: Eight instruction-tuned decoder-only transformer models (ranging from 7B to 14B parameters), including Falcon3, Gemma-2, Llama-3.1, Mistral variants, Phi-4, Qwen2.5, and rnj-1.
• Vocabulary: A shared set of 5,000 high-frequency English nouns derived from the SUBTLEX-US lexicon and the C4 corpus.
• Scale: The study collected over 17.5 million trials across the models.

Behavioral Paradigms

Two psycholinguistic paradigms were used to generate behavioral similarity matrices ($S_{beh}$):

1. Forced Choice (FC): The model is given a cue word and a list of 16 candidate words, selecting the 2 most semantically related. This creates a dense, constrained cue-response matrix ($B_{FC}$).
2. Free Association (FA): The model is given a cue word and asked to generate 5 distinct associated words. This creates a sparser, open-ended cue-response matrix ($B_{FA}$).

• Processing: Raw counts were reweighted using Positive Pointwise Mutual Information (PPMI) to reduce frequency bias. Behavioral similarity matrices were computed via cosine similarity between the PPMI-weighted row vectors.

Hidden-State Extraction

Hidden-state representations ($h_\ell$) were extracted for the same vocabulary across all layers ($\ell$) using four distinct contextual strategies:

1. Averaged: Word embedded in 50 natural sentences from the C4 corpus (averaged).
2. Meaning: Word embedded in a definition-style prompt ("What is the meaning of {w}?").
3. Task (FC): Word embedded in the FC prompt (without candidates).
4. Task (FA): Word embedded in the FA prompt.

• Preprocessing: Vectors were mean-centered to mitigate anisotropy (the "cone effect" in transformer spaces) before computing cosine similarity matrices ($S_{hid}^\ell$).

Evaluation Metrics

1. Representational Similarity Analysis (RSA): Pearson correlation between the pairwise similarities of the hidden-state matrix and the behavioral/reference matrices.
2. Nearest-Neighbor Overlap (NN@k): The fraction of shared top-$k$ neighbors between hidden states and behavioral spaces.
3. Held-Out-Word Ridge Regression: A predictive task where hidden-state similarities for unseen word pairs are predicted using behavioral similarities (FC/FA) alongside lexical baselines (FastText, BERT) and a cross-model consensus baseline.

3. Key Contributions

1. Behavior-Activation Alignment Framework: The authors provide the first systematic comparison of LLM behavioral geometry (derived from FC and FA) against internal layer-wise hidden-state geometry using RSA and nearest-neighbor overlap.
2. Superiority of Forced Choice: They demonstrate that Forced Choice (FC) behavior aligns significantly better with internal hidden-state geometry than Free Association (FA). FC yields higher RSA correlations and better predictive power in regression tasks.
3. Context-Dependent Alignment: The study reveals that the layer at which behavior best matches internal states depends on the extraction strategy. Task-aligned and Meaning prompts show peak alignment in earlier/mid-layers, while Averaged (natural context) strategies shift peak alignment to deeper layers.
4. Recoverability of Internal States: Using ridge regression, the paper proves that behavioral similarity (specifically FC) predicts unseen hidden-state similarities beyond what is achievable with lexical baselines (FastText/BERT) or cross-model consensus, indicating that discrete outputs retain recoverable information about internal semantic organization.
5. Open Data Release: The authors released a large-scale dataset of 17.5M+ association trials and the full codebase for reproducibility.

4. Key Results

• RSA Correlations:
  • FC vs. Hidden States: Mean RSA correlation for FC ranged from 0.346 (Averaged strategy) to 0.463 (Task FC strategy).
  • FA vs. Hidden States: FA showed much weaker alignment, ranging from 0.140 to 0.199.
  • Baselines: Cross-model consensus showed the highest correlation overall ($r \approx 0.79$), confirming a shared "platonic" semantic subspace across models, but behavioral FC still provided unique, non-trivial signal.
• Predictive Power (Ridge Regression):
  • Adding FC behavioral features to a baseline (FastText + BERT + Cross-Model) improved the $R^2$ score for predicting hidden-state similarity by +0.022 on average.
  • FA provided negligible gains (+0.002).
  • Peak performance reached $R^2 = 0.844$ for Llama-3.1-8B-Instruct under the "Meaning" strategy.
• Layer Dynamics:
  • FC alignment peaks in mid-layers (approx. layers 10–12) when using task-aligned prompts, suggesting these layers encode the core semantic constraints required for the task.
  • FA alignment is generally weaker and noisier, likely due to the sparsity and heavy-tailed nature of free association responses.

5. Significance and Implications

• Cognitive Science Validation: The study validates the core assumption of cognitive science that structured behavior is constrained by internal states. It proves that for LLMs, "black-box" behavioral probes can partially reconstruct the "white-box" internal geometry.
• Methodological Guidance: The findings suggest that constrained behavioral tasks (Forced Choice) are superior measurement instruments for probing internal semantic structure compared to open-ended tasks (Free Association). The controlled candidate sets in FC concentrate probability mass, yielding a higher signal-to-noise ratio.
• Interpretability & Safety: The results imply that even without access to logits or weights, one can infer significant aspects of a model's internal semantic organization. This has implications for model auditing, detecting biases, and understanding how models organize knowledge.
• Universal Geometry: The strong cross-model consensus baseline supports the hypothesis of a shared, low-dimensional semantic subspace across diverse LLMs, while the behavioral results show how specific model behaviors map onto this shared structure.

In conclusion, the paper establishes that behavioral geometry is a recoverable proxy for internal semantic geometry, provided the behavioral task is sufficiently constrained (like Forced Choice) to minimize noise and maximize the alignment between observable outputs and latent representations.