Technical Summary: Observable Patterns Are Not Explanations: A Causal-Geometric Analysis of Latent Reasoning Models

Problem Statement

Latent Reasoning Models (LRMs) replace explicit Chain-of-Thought (CoT) traces with continuous, hidden-state computations to improve efficiency. However, this shift removes the human-interpretable artifacts necessary for monitoring and oversight. Recent interpretability efforts have attempted to decode these latent states, identifying observable patterns such as Breadth-First-Search (BFS)-like frontiers in graph tasks and decodable arithmetic "scratchpads" in reasoning tasks.

The core problem addressed is that these observational patterns may be epiphenomenal. Prior work often treats the mere presence of decodable structure or static geometric patterns as evidence of internal reasoning mechanisms. This paper argues that such approaches risk conflating decodability with mechanism, potentially leading to "false explanations" (analogous to the "dead salmon" effect in neuroimaging) where patterns appear explanatory but do not causally drive model behavior. The authors posit that without rigorous causal testing and matched controls, observable patterns cannot establish that a model is actually reasoning.

Methodology

The authors employ a causal-first framework, shifting the focus from observational proxies to the causal influence of latent thoughts on model behavior. The study involves:

1. Experimental Setup & Controls:

   • Tasks: Graph-hopping (ProsQA) and Arithmetic Reasoning (GSM8k).
   • Target Models: Coconut (Hao et al., 2025) and CODI (Shen et al., 2025), representing two distinct LRM training paradigms (curriculum-based recurrence and self-distillation).
   • Matched Controls: To isolate specific mechanisms, the authors introduce:
     • Recurrence Control (PaT): Follows Coconut's curriculum but removes recurrent feedback, replacing it with learned static tokens.
     • Curriculum Control (Cu): A Coconut variant with a perturbed curriculum (randomized stage sampling) but preserved recurrence.
     • Observational Controls: Base GPT-2 and Explicit-CoT GPT-2 to test if patterns emerge without latent training.

2. Causal Interventions:

   • Thought Ablation: Removing latent thought positions at test-time to assess necessity.
   • Causal Tracing: Extending activation patching (Meng et al., 2022) to latent positions. The authors measure the Indirect Effect (IE) by injecting clean activations from a target prompt into a corrupted partner prompt to see if the model recovers its correct behavior.
   • Gradient-Subspace Interventions: Instead of treating thought influence as binary, the authors identify low-rank directions in the latent space most sensitive to the loss (via SVD of gradients). They then ablate or amplify these specific subspaces to measure graded causal influence.

3. Geometric & Dynamic Analysis:

   • Markovianity: Testing if future latent states can be predicted from previous ones using linear and MLP regressors.
   • Subspace Stability: Measuring the geometric alignment (cosine similarity of principal angles) of the causal gradient subspaces across time steps.

Key Results

1. Observable Patterns Are Not Unique to LRMs (RQ1)

• Graph-Hopping (BFS/Superposition): The "BFS-like" pattern (rising probability of correct nodes, falling entropy) and "superposition" (holding multiple candidates) observed in Coconut were also reproduced by the PaT control (which lacks recurrence) and, to a lesser extent, by the Explicit-CoT control. Conversely, the curriculum-perturbed control (Cu) inverted these patterns despite retaining recurrence.
• Arithmetic (Scratchpad): Logit-lens decoding of intermediate arithmetic steps in CODI was also observed in PaT and other controls. Step-alignment decayed sharply with depth for all models, contradicting the hypothesis of an ordered scratchpad.
• Conclusion: Observable patterns like BFS frontiers or scratchpad alternation are not sufficient evidence for specific reasoning mechanisms (recurrence or distillation) as they appear in models lacking those mechanisms.

2. Latent Thought Influence is Graded and Localized (RQ2)

• Ablation vs. Causality: While some models (e.g., Coconut on graph-hopping) maintained performance when thoughts were removed, causal tracing revealed that latent thoughts still exert a graded causal influence.
• Localization: The influence of latent thoughts is not distributed uniformly across the representation. Instead, it concentrates in low-rank, loss-sensitive directions (the gradient subspace).
• Task Dependence: On arithmetic reasoning, ablating these gradient subspaces caused significant performance drops, whereas on graph-hopping, the influence was weaker but still measurable and capable of steering predictions when amplified.

3. Dynamics of Causal vs. Non-Causal Directions (RQ3)

• Static vs. Evolving: Full latent thought trajectories often appear static (highly predictable by identity or simple linear maps), especially in tasks where causal influence is weak (e.g., graph-hopping).
• Causal Subspace Dynamics: However, the specific gradient subspaces where causal influence concentrates exhibit more structured, non-trivial evolution. In arithmetic reasoning, these subspaces rotate and evolve significantly, whereas in graph-hopping, they remain geometrically stable.
• Takeaway: The "reasoning" dynamics are not visible in the full latent trajectory but are concentrated in specific, behavior-influential subspaces.

Significance and Claims

The paper claims to shift the paradigm of LRM interpretability from observational to causal.

• Hidden Computation, Not Hidden Explanation: The authors argue that latent thoughts should be treated as "hidden computation" rather than "hidden explanations." The presence of decodable structure or geometric patterns does not constitute a mechanism unless it is shown to causally drive behavior.
• Methodological Rigor: The work demonstrates that claims about latent reasoning mechanisms require matched controls (to rule out curriculum or formatting artifacts) and causal tests (to distinguish between correlation and causation).
• Graded Influence: The binary view of "thoughts used vs. bypassed" is insufficient. Influence is a continuous quantity, localized to specific low-rank subspaces that vary by task and model.
• Safety Implications: Relying on observational proxies risks "false interpretability," where auditors monitor the wrong features. A causal-first approach provides a more grounded basis for auditing agentic systems, ensuring that safety monitoring targets the actual computational circuits driving behavior.

The authors conclude that while latent reasoning models offer efficiency, their interpretability requires a stricter evidential standard that prioritizes causal localization over the mere decoding of latent states.