Technical Summary: Automated Interpretability and Feature Discovery in Language Models with Agents

Problem Statement

Mechanistic interpretability aims to understand the internal representations of large language models (LLMs) to determine how specific concepts, behaviors, or safety properties are encoded. While recent advances like Sparse Autoencoders (SAEs) have decomposed dense activations into sparse, interpretable features, existing automated interpretability methods suffer from two primary limitations:

1. Brittleness of One-Shot Explanations: Current automated approaches typically generate a single natural-language description of a feature based on its top-activating examples. These "one-shot" labels often rely on limited evidence, fail to account for specific constraints (e.g., linguistic or structural), and are prone to being falsified by counterexamples.
2. The Discovery Bottleneck: Most workflows assume the feature of interest is already known. In realistic scenarios, users begin with high-level questions (e.g., "find features related to a specific language" or "find safety-relevant features") without knowing which of the thousands of available features to inspect. Systematic, unsupervised feature discovery remains a significant, under-addressed bottleneck in the field.

Methodology: InterpAgent

The authors introduce InterpAgent, an autonomous multi-agent framework that unifies feature discovery and hypothesis refinement into a single, iterative empirical loop. The system operates via a supervisor coordinating two specialized sub-agents within a shared execution environment:

1. FeatureFinder (Statistical Discovery)

This agent addresses the discovery bottleneck. Given a set of prompts labeled by high-level categories (e.g., language, topic, or safety), it identifies candidate features that reliably differentiate these groups.

• Process: It constructs an activation matrix from the model's internal representations, applies PCA for dimensionality reduction, and builds a k-nearest-neighbor (k-NN) graph in the embedded space.
• Clustering & Testing: Using graph community detection (Leiden algorithm), it clusters prompts. It then performs statistical testing (Wilcoxon rank-sum test with Benjamini–Hochberg correction) to find features enriched in specific clusters.
• Filtering: Candidates are filtered by adjusted significance, effect size (Cohen's d), and log-fold change to ensure practical relevance, outputting a ranked list of feature IDs with supporting evidence.

2. FeatureExplainer (Iterative Hypothesis Refinement)

This agent transforms a candidate feature into a robust, auditable explanation through a "propose–test–revise" loop.

• Hypothesis Generation: It starts with seed hypotheses (from baselines or top activations) and maintains competing hypotheses.
• Stress-Testing: For each hypothesis, the agent generates contrastive prompt suites (predicted positives and matched negative controls) and runs them through the target model to record activations.
• Multi-Metric Evaluation: Hypotheses are scored on a battery of seven metrics:
  1. Detection F1 (discriminability)
  2. Fuzzing F1 (robustness to token perturbations)
  3. Surprisal AUROC (coherence)
  4. Embedding Similarity (semantic alignment)
  5. Statistical Separability (p-value)
  6. Effect Size (Cohen's d)
  7. LLM-as-Judge Coherence
• Refinement: The agent ranks hypotheses, identifies failure modes in the weakest metrics, and proposes targeted revisions. The loop continues until convergence or a fixed budget is reached.
• Polysemanticity Detection: If multiple semantically distinct hypotheses achieve comparable support, the feature is flagged as polysemantic, and all competing interpretations are reported rather than forcing a single label.

Key Contributions

1. Unified Framework: The introduction of InterpAgent, which combines unsupervised feature discovery and iterative hypothesis refinement into a single natural-language-driven loop, producing auditable traces for every step.
2. Statistical Discovery Algorithm: A novel pipeline (FeatureFinder) that uses k-NN graph clustering and differential activation analysis to retrieve category-specific marker features without prior knowledge of feature IDs.
3. Iterative Refinement Pipeline: A rigorous explanation process (FeatureExplainer) that utilizes a 7-metric evaluation battery to sharpen hypotheses, detect polysemanticity, and produce falsifiable explanations.
4. Generalizability: Demonstration that the framework applies not only to SAE features but also directly to raw MLP neuron activations in weight-sparse transformers, recovering manually identified circuits without SAEs.

Results

The framework was evaluated on the Gemma-2 family of models and weight-sparse transformers.

• Refinement Performance: Iterative refinement consistently outperformed one-shot auto-interpretations. On the Gemma-2 family, refined hypotheses were selected as the best explanation for approximately 80% of features (using Gemini 2.5 Pro and Claude 4 Sonnet) and 64% (using GPT-4o-mini). The improvements primarily corrected missing constraints and over-broad labels within the first one or two iterations.
• Discovery Accuracy: The FeatureFinder algorithm successfully recovered language-specific features (across 20 languages) and non-linguistic categories (coding, math, safety) with high statistical significance. The k-NN graph approach significantly outperformed standard clustering baselines (K-Means, Agglomerative) in aligning activation space with language identity.
• Case Studies:
  • Monosemantic Convergence: The agent successfully refined a vague "technology knowledge" label into a specific "informal French syntax in tech troubleshooting" explanation.
  • Polysemanticity Detection: The system correctly identified a feature with competing "coding/equations" and "technical-scientific discourse" hypotheses, flagging it as polysemantic rather than forcing a single label.
  • Safety Auditing: The agent discovered a feature related to "explosives," validated it, and demonstrated that causal intervention (negative activation steering) reduced refusal behavior on safety-sensitive prompts, confirming the feature's functional role in the refusal circuit.
• Raw Neuron Discovery: Applied to raw MLP neurons in a weight-sparse transformer, the agent automatically rediscovered neurons previously identified by human experts through manual circuit analysis, achieving over 75% mean validation accuracy.

Significance and Claims

The paper argues that interpretability should be treated not as a static labeling task but as an experimental process involving hypothesis proposal, testing, and revision. By automating this workflow, InterpAgent produces explanations that are:

• Sharper and More Falsifiable: Moving beyond brittle one-shot labels to explanations validated against counterexamples and quantitative metrics.
• Auditable: Providing a complete trace of hypotheses, test cases, metrics, and interventions.
• Scalable: Capable of handling the discovery of features from scratch, addressing the bottleneck of identifying which features to explain in large models.

The authors claim that this agent-driven approach yields more robust insights than current automated methods and highlights the potential for systematic auditing of feature libraries at scales currently impractical for manual analysis. They note that the effectiveness of the system scales with the capabilities of the underlying LLM used for hypothesis generation and refinement.