Technical Summary: Towards Effective Theory of LLMs: A Representation Learning Approach

Problem Statement

Large language models (LLMs) are increasingly deployed in critical domains, yet their internal computations remain opaque. While the microscopic operations of LLMs (matrix multiplications, attention updates, activations) are known, understanding behavior directly from these details is often intractable and fails to reveal the macroscopic organization necessary for explanation. Current interpretability methods, such as tracing circuits or using sparse autoencoders (SAEs), often struggle to capture coherent, high-level computational phases or "mental states" that persist across token sequences. There is a need for a framework that abstracts away irrelevant microscopic details while preserving the structure required for prediction, interpretation, and intervention, analogous to effective theories in physics (e.g., thermodynamics) which describe complex systems via macrovariables.

Methodology: Representational Effective Theory (RET)

The authors propose Representational Effective Theory (RET), a framework that learns macrostates from LLM hidden-state trajectories using a self-supervised representation learning approach.

Core Framework

RET treats the LLM's hidden states as a microscopic substrate ($x_t$) and learns a mapping to a lower-dimensional macrostate ($z_t$) that satisfies three desiderata:

1. Abstraction from Microscopic Detail: Discarding token-local variation while preserving behaviorally meaningful structure.
2. Approximately Closed Dynamics: The next macrostate ($z_{t+1}$) should be predictable from the current macrostate ($z_t$) with minimal reliance on the full microstate ($x_t$).
3. Practical Relevance: The macrostates must support concrete interpretability tasks (interpretation, prediction, intervention).

Learning Procedure

• Data Substrate: Hidden-state prefixes ($h^{(\ell_{mid})}_{0:t}$) are extracted from a fixed middle layer of a frozen LLM.
• Architecture: A lightweight encoder ($f_\theta$) maps the prefix to a macrostate $z_t$. A predictor ($T_\phi$) forecasts the next macrostate $z_{t+1}$ from $z_t$.
• Objective: The system is trained using a BYOL/JEPA-style self-supervised objective. The predictor minimizes the $L_1$ distance between its prediction and the target encoder's output for the next step:
  $$L_{pred} = \|T_\phi(f_\theta(x_t)) - \bar{f}_\theta(x_{t+1})\|_1$$
  where $\bar{f}_\theta$ is an Exponential Moving Average (EMA) of the encoder. This objective encourages the encoder to compress the microstate into a representation that supports autonomous prediction, effectively learning an "effective theory" of the model's dynamics.
• Clustering: Learned macrostates are clustered (e.g., via K-means) and aggregated into discrete groups to form a symbolic state machine representing the model's "mental state."

Key Contributions

1. Formulation of an Effective Theory Perspective: The paper defines a principled view of LLM interpretability centered on macrovariables, approximate closure, and practical relevance, drawing parallels to physics and complexity science.
2. RET Framework: Introduction of a simple representation-learning method to acquire macrostates and their transition laws from hidden-state trajectories without task labels.
3. Empirical Validation: Demonstration that RET macrostates are useful across three distinct interpretability settings:
   • Interpretation: Generating coherent, temporally stable trajectories of reasoning phases.
   • Prediction: Early detection of downstream behavioral outcomes (specifically sycophancy).
   • Intervention: Providing causal handles to steer model generation toward specific computational strategies.

Experimental Results

1. Interpretable Reasoning Trajectories

• Setup: RET was applied to GPT-OSS-20B solving NuminaMath problems. Macrostates were clustered into 12 groups.
• Findings: RET produced temporally consistent trajectories that aligned with human-interpretable reasoning phases (e.g., problem setup, symbolic transformation, verification).
• Comparison: Baselines (raw hidden states, PCA, SAE features) produced unstable, token-level switching trajectories.
• Metric: The Mental-state Interpretability Score (MIS) was introduced. RET achieved a score of 5.61 on held-out data, significantly outperforming SAEs (4.24), indicating RET better recovers the reasoning narrative visible in raw text.

2. Temporal Consistency

• Setup: Experiments on Pythia-160M with planted scene changes (e.g., soccer game $\to$ rainstorm $\to$ library).
• Findings: RET macrostates ($z_t$) exhibited smooth trajectories with sharp discontinuities at scene boundaries, forming block-diagonal structures in cosine similarity matrices.
• Comparison: Baselines (raw $h$, pooled $h$, SAE) remained dominated by fast lexical fluctuations and failed to preserve context boundaries or distinguish repeated words in different contexts.

3. Early Sycophancy Prediction

• Setup: A new multi-turn sycophancy dataset (SYCON-Bench extended to ~14k samples) was constructed. Probes were trained to predict if a model would eventually "cave" to false presuppositions or debate pressure, using only the first 5–10% of the response.
• Findings: A supervised variant of RET (trained with an auxiliary sycophancy loss) achieved the highest balanced accuracy (0.80 in False Presupposition, 0.90 in Debate), outperforming all hidden-state baselines (including a Transformer Block baseline) and unsupervised RET.
• Implication: The learned macrostates capture behaviorally relevant structure that allows for early detection of alignment failures.

4. Unsupervised Steering

• Setup: The model's hidden states were nudged toward or away from specific cluster centers (attractors/repulsors) during generation.
• Findings: Steering successfully shifted solution strategies. For example, steering toward a "known formula retrieval" cluster caused the model to use closed-form formulas instead of sequential enumeration. Steering toward "wording disambiguation" resolved ambiguous currency units.
• Limitations: Steering is effective only among computational paths already plausible under the prompt; it cannot induce behaviors that are effectively unavailable in the model's distribution (e.g., manually summing 61 integers when the model defaults to a formula).
• Comparison: RET steering achieved 91% coverage of behavioral shifts across 33 diverse prompts, outperforming heavily tuned SAE-baseline steering (61% coverage).

Significance and Claims

The paper claims that LLM computation admits useful effective descriptions via RET. The authors argue that:

• Macro-level variables learned via predictive self-supervision can abstract away microscopic noise to reveal coherent "mental states."
• These states are not merely descriptive but are actionable, providing causal handles for steering model behavior.
• The approach offers a complementary perspective to mechanistic interpretability (which focuses on sparse circuits) by focusing on trajectory-level, coarse-grained dynamics.

The authors remain modest, noting that RET is one instantiation of an effective theory and that the results are specific to the tested settings (mathematical reasoning, sycophancy detection). They emphasize that the "mental state" is an operational term for a coarse computational phase, not a literal cognitive state, and that the symbolic labels are useful summaries derived via LLM-assisted clustering rather than definitive ground truth.