Technical Summary: Building Better Activation Oracles

Problem Statement

Activation Oracles (AOs) are fine-tuned Large Language Models (LLMs) designed to interpret the residual stream activations of a target LLM by answering natural language questions about the model's internal state. While promising for scalable, end-to-end interpretability, current AOs suffer from significant usability issues:

• Hallucinations: The oracle outputs false information about the model's reasoning.
• Vagueness: Outputs are generic and unfalsifiable, failing to answer specific user questions.
• Evaluation Challenges: "Text-inversion confounds" allow models to appear performant by simply reconstructing surrounding text from activations rather than truly interpreting the internal state.
• Dataset and Objective Flaws: The standard training pipeline relies on the LatentQA dataset and a FineWeb based past/future-lens objective, both of which the authors identify as suboptimal for training robust AOs.

Methodology

The authors propose a revised training regime for AOs involving four specific improvements to the standard recipe established by Karvonen et al. [2025].

1. Improved Conversational Dataset

The authors replace the LatentQA dataset with a newly constructed conversational dataset designed to mitigate text-inversion and vagueness.

• Construction: A separate LLM (Sonnet 4.6) is provided with a target model's Chain-of-Thought (CoT). It splits the CoT into a prefix and a suffix, then generates a question about the suffix.
• Constraint: The question is constructed to be difficult to answer using only the prefix text (preventing text-inversion) but solvable using the activations at the split point.
• Rationale: This targets the model's internal reasoning rather than superficial prompt specifics and avoids incentivizing guessing.

2. Multi-Layer Activation Feeding

Instead of feeding activations from a single randomly selected layer (e.g., 25%, 50%, or 75% depth), the authors experiment with feeding activations from multiple contiguous layers.

• Findings: Performance peaks when feeding activations from layers 21–25 (centered around layer 22, or ~62% depth).
• Benefit: This provides a consistent uplift, particularly on model diffing tasks, though a single well-chosen layer (65% depth) captures most of the gain.

3. On-Policy Training Data

The authors replace the pre-training corpus (FineWeb) used for the past/future-lens objective with on-policy data (generations from the model being interpreted).

• Hypothesis: On-policy data is a more solvable task because it targets what the model is actually representing in its activations during generation, rather than relying on static pre-training priors.
• Application: This aligns the training distribution with the intended use case (e.g., studying agent traces).

4. Injection Formula and Strength

The authors refine how activations are injected into the AO.

• Method: They maintain the additive, norm-matched injection after the second transformer layer (following Karvonen et al. [2025]) rather than replacing token embeddings entirely (as in Natural Language Autoencoders).
• Hyperparameter Tuning: They found that increasing the magnitude of the injection strength marginally improves performance. This is hypothesized to help overcome the language model's prior to weight tokens equally, enforcing a stronger signal for the activation.

Key Contributions

1. AObench: The release of the first comprehensive evaluation suite for AO quality. It is designed to measure specific capabilities (concreteness, faithfulness) while remaining robust to text-inversion confounds.
2. Revised Training Recipe: A demonstration that specific changes to the dataset, layer selection, and training data distribution yield consistent improvements in AO quality.
3. Open Source Resources: Release of the new conversational dataset, model checkpoints, and evaluation code.

Results

The authors conducted controlled ablation studies starting from the Karvonen et al. [2025] baseline, training all models on exactly 50M tokens.

• Overall Performance: The full recipe improved the chance-adjusted AObench score from +0.244 (baseline) to +0.435.
• Primary Driver: The swap to the new conversational dataset was the single largest contributor, raising the score from +0.244 to +0.310.
• Secondary Drivers: Multi-layer extraction and on-policy data provided additional, smaller uplifts. Increasing injection strength provided a final marginal gain.
• Qualitative Improvements:
  • Vagueness: Improved substantially (chance-adjusted score from 0.076 to 0.205).
  • Hallucination: The raw hallucination score initially appeared to increase because the new dataset trained the model to make more specific claims (making hallucinations easier to detect). However, when accounting for this specificity, the rate of unsupported details decreased monotonically (from 68.8% to 84.6% faithfulness).
• Negative Finding: Narrow post-training on tasks from Ivanova et al. [2026] consistently failed to exceed simple linear-probe performance.

Significance and Claims

The paper positions AOs as a critical component of the emerging paradigm of scalable, end-to-end interpretability, where models are trained on self-supervised objectives to map internal states to natural language.

• Modest Capability Claims: The authors note that while "quality of life" improvements (reduced vagueness, better instruction following) are substantial, the raw capability improvements are "marginal."
• Future Headroom: The authors argue that the largest potential gains for AOs will likely come from improvements to the unsupervised training task itself. They view their current work as a foundational step, suggesting that finding a scalable task that significantly increases capability remains a major open challenge.
• Evaluation Standard: By introducing AObench, the paper aims to provide a stable, robust metric for future research, addressing the difficulty of evaluating AOs against text-inversion confounds.