Levels of Analysis for Large Language Models

1. Problem Statement

The rapid advancement of Large Language Models (LLMs) and related AI systems has created a significant interpretability crisis. These models, built on massive artificial neural networks with billions of parameters, are increasingly powerful yet opaque. Computer scientists face a unique challenge: they have created systems they do not fully understand, and for many proprietary models, internal weights and training data are inaccessible. Consequently, researchers are forced to infer the nature of these systems solely through their external behavior.

The authors argue that this problem mirrors the historical challenge faced by cognitive scientists in understanding the human mind. Just as cognitive science developed methods to study minds without direct access to neural circuitry, these same methods can be applied to AI. The paper proposes that David Marr's three levels of analysis (Computational, Algorithmic, and Implementation) provide a necessary framework to systematically understand LLMs, moving beyond simple performance benchmarks to uncover the underlying mechanisms, representations, and limitations of these "new kinds of minds."

2. Methodology: Marr's Levels of Analysis Applied to AI

The paper structures its analysis by mapping established cognitive science techniques onto the three levels of analysis proposed by David Marr:

A. Computational Level (The "What" and "Why")

• Focus: The abstract problem the system solves and the ideal solution.
• Methodology: The authors analyze LLMs based on their explicit training objectives (e.g., next-token prediction/autoregression) and compare their behavior to optimal solutions derived from Bayesian inference or axiomatic systems (e.g., probability theory, decision theory).
• Key Techniques:
  • Rational Analysis: Predicting behavior based on the constraints of the environment and the training objective.
  • Bayesian Modeling: Treating LLMs as approximate Bayesian reasoners to infer their implicit priors.
  • Axiomatic Testing: Checking for coherence against logical rules (e.g., $P(A) + P(\neg A) = 1$).

B. Algorithmic Level (The "How")

• Focus: The representations and algorithms used to approximate the solution.
• Methodology: Since LLM internal algorithms are learned and opaque, the authors adapt cognitive psychology techniques that infer internal structures from behavioral patterns.
• Key Techniques:
  • Parallel vs. Serial Processing Analysis: Examining error patterns (e.g., "illusory conjunctions" in vision-language models) to determine if the model processes information in parallel (prone to interference) or serially.
  • Similarity Judgments & Multidimensional Scaling (MDS): Prompting models to rate the similarity of stimuli (e.g., colors, sounds, concepts) to reconstruct their internal representational geometry.
  • Implicit Association Tests (IAT): Adapting psychological tests to bypass safety alignments and reveal hidden biases and associations within the model's weights.

C. Implementation Level (The "Physical Realization")

• Focus: How representations and algorithms are physically realized in the hardware (neural network weights and units).
• Methodology: Drawing parallels with neuroscience, the authors utilize mechanistic interpretability tools to map internal states to functions.
• Key Techniques:
  • Representational Analysis: Correlating activation patterns with specific concepts (e.g., spatial/temporal encoding).
  • Causal Interventions: Using Activation Patching (replacing activations to test function), Sparse Autoencoders (SAEs) to identify interpretable features, and Concept Activation Vectors (CAVs) to manipulate specific behaviors causally.

3. Key Contributions and Results

Computational Level Insights

• "Embers of Autoregression": The authors demonstrate that LLM behavior is heavily influenced by their training objective (next-token prediction). Even in deterministic tasks (like counting or decoding ciphers), LLMs perform significantly better when the correct answer is a high-probability token sequence than a low-probability one. This suggests that "reasoning" is often just high-probability text generation.
• Bayesian Priors: LLMs can be modeled as Bayesian agents. By analyzing their outputs, researchers can recover the implicit prior distributions the models have learned about the world (e.g., duration of events, causal relationships).
• Systematic Violations: LLMs exhibit systematic violations of probability axioms (e.g., incoherent probability judgments) and logical rules, mirroring human cognitive biases. This highlights a distinction between competence (underlying ability) and performance (observed behavior).

Algorithmic Level Insights

• Parallel Processing Interference: Vision-Language Models (VLMs) exhibit "pop-out" search capabilities but degrade in "conjunction search" tasks as distractors increase, similar to human visual working memory limits. This suggests VLMs rely on compositional representations processed in parallel, leading to feature interference.
• Sensory Representations: Despite being trained only on text, LLMs encode rich sensory representations. Similarity judgments reveal that LLMs organize concepts (colors, pitch, consonants) in geometric structures (e.g., color wheels, pitch helices) that closely match human perceptual spaces.
• Uncovering Hidden Biases: Standard safety alignments (RLHF) often mask biases in direct queries. However, implicit association tests reveal that models still hold strong internal associations (e.g., linking "Julia" with domestic roles and "Ben" with management), which influence downstream decisions.

Implementation Level Insights

• Mechanistic Understanding: The paper validates that specific circuits (e.g., "induction heads") and distributed features (identified via SAEs) causally implement capabilities like in-context learning and factual recall.
• Causal Validation: Techniques like activation patching confirm that specific neural activations are not just correlated with but are causally responsible for specific model behaviors.

Beyond Marr: Limits of AI

• Adversarial Task Design: By understanding the algorithmic constraints of LLMs (e.g., parallel processing limits), researchers can design tasks that are uniquely difficult for AI but easy for humans (and vice versa).
• Reasoning Trade-offs: Contrary to the assumption that "more reasoning" (e.g., Chain-of-Thought) is always better, the authors show that for certain tasks (implicit statistical learning, face recognition), forcing verbal reasoning actually degrades LLM performance, mirroring human "verbal overshadowing" effects.

4. Significance

• Interdisciplinary Framework: The paper provides a rigorous, structured framework for the AI interpretability community, moving beyond ad-hoc analysis to a systematic application of cognitive science.
• New Toolkit: It introduces a "toolkit" of cognitive science methods (MDS, IAT, reaction time proxies, causal patching) specifically adapted for probing black-box AI systems.
• Reframing AI Evaluation: It argues that evaluating AI should not just focus on task success rates but on understanding the mechanisms of failure and success. This helps distinguish between genuine general intelligence and artifacts of training objectives.
• Bridging Human and Artificial Minds: The findings suggest deep structural similarities between human cognition and LLMs (e.g., parallel processing limits, probabilistic biases), suggesting that LLMs are not just statistical engines but systems that develop internal models of the world similar to biological minds, albeit with different constraints.

In conclusion, the paper posits that by treating LLMs as information processing systems subject to the same analytical levels as human minds, researchers can demystify their behavior, identify their true capabilities and limitations, and guide the development of more robust and interpretable AI.