Interpreting Language Models Through Concept Descriptions: A Survey

Based on the survey paper "Interpreting Language Models Through Concept Descriptions: A Survey" by Feldhus and Kopf, here is a detailed technical summary covering the problem, methodology, contributions, results, and significance.

1. Problem Statement

The central challenge in Mechanistic Interpretability is reverse-engineering the internal algorithms of Large Language Models (LLMs) to understand their decision-making processes. While researchers can identify specific model components (e.g., neurons, attention heads) or learned abstractions (e.g., Sparse Autoencoder features), assigning human-understandable meaning to these components remains difficult.

• Limitations of Early Approaches: Traditional methods relied on probing classifiers or testing alignment with pre-defined, fixed sets of concepts (e.g., specific syntactic roles or semantic categories). These approaches are limited because they cannot discover novel, emergent concepts that the model has learned but were not anticipated by human researchers.
• The Polysemanticity Challenge: Individual neurons often exhibit polysemanticity, activating for diverse and seemingly unrelated inputs due to the phenomenon of superposition (encoding many concepts as linear combinations of neuron activations). This makes single-neuron descriptions often vague or inaccurate.
• The Gap: There is a lack of a systematic overview regarding how to generate open-vocabulary, natural language concept descriptions for model internals using the generative power of LLMs themselves, and how to rigorously evaluate the quality of these descriptions.

2. Methodology and Framework

The paper surveys a rapidly emerging field where LLMs act as "interpreters" to generate natural language explanations for the components of other models. The methodology is structured around three core pillars:

A. Targets of Interpretation

The survey categorizes the targets of description into two groups:

1. Native Model Components:
   • Neurons: Individual units in Feed-Forward Networks (FFNs).
   • Attention Heads: Components responsible for processing relational patterns between tokens.
2. Model Abstractions:
   • SAE Features: High-dimensional, sparsely activated features extracted via Sparse Autoencoders, designed to be more monosemantic (representing a single concept) than raw neurons.
   • Circuits: Subgraphs of interacting components that perform specific tasks (e.g., Indirect Object Identification).

B. Generation Methods

The dominant paradigm involves Auto-interpretability:

1. Data Collection: Identify "maximally activating" input samples (text fragments) for a specific component or feature.
2. Prompting: Feed these samples to a powerful generator LLM (e.g., GPT-4, Claude) with a prompt asking for a description of the concept the component detects.
3. Output: The generator produces a natural language summary (e.g., "legal clauses," "references to the 1980s," or "subject-verb agreement").

C. Evaluation Techniques

The paper synthesizes a diverse toolkit for evaluating these descriptions, categorized into five families:

1. Predictive Simulation: Using a "simulator" LLM to predict feature activations based on the generated description. High correlation implies the description captures the feature's logic (though this is limited by the simulator's own opacity).
2. Input-Based Evaluation: Measuring purity (does the feature only fire for the described concept?) and coverage (does it fire for all instances of the concept?) using metrics like AUROC and mean activation differences.
3. Output-Based Evaluation (Causal): Testing faithfulness via interventions. If a feature is artificially activated, does the model's output shift in the direction predicted by the description?
4. Semantic Similarity: Comparing the generated description against ground-truth labels using embedding models (cosine similarity).
5. Human Evaluation: Assessing correctness, plausibility, readability, and faithfulness through human annotation.

3. Key Contributions

This survey provides the first comprehensive overview of the field of concept descriptions for model components. Its specific contributions include:

• Taxonomy of Methods: A structured classification of techniques for generating descriptions for neurons, attention heads, SAE features, and circuits.
• Evaluation Landscape: A detailed mapping of evaluation metrics, highlighting the shift from simple correlation scores to multi-faceted, causal, and human-centric assessments.
• Identification of Trends:
  • A move away from raw neurons toward SAEs to mitigate polysemanticity.
  • The emergence of frameworks that handle multiple concepts per feature.
  • The maturation of evaluation from "does it correlate?" to "is it causally faithful?"
• Roadmap for Future Research: The authors identify critical gaps and propose specific directions for advancement.

4. Key Results and Findings

• Effectiveness of SAEs: Research indicates that Sparse Autoencoders (SAEs) successfully decompose polysemantic neurons into more interpretable, monosemantic features, making them the preferred target for concept description in recent work.
• Limitations of Simulation: While "simulator correlation" is the most common automated metric, it has low precision (often <0.8 correlation) and introduces a layer of abstraction (using one black box to explain another).
• Causal Faithfulness is Rare: Most current descriptions are correlational. Rigorous causal evaluation (via intervention/steering) is difficult and less common, yet essential for verifying that a description reflects the actual function of a component.
• Human Bias in Interpreters: The LLMs used to generate descriptions may have their own biases, potentially favoring concrete, atomic concepts (e.g., "Paris") over abstract relational concepts (e.g., "syntactic agreement").
• Data Constraints: The concepts a model can represent are bounded by its training data. Descriptions cannot reveal concepts absent from the training corpus, creating a potential "interpretability illusion."

5. Significance and Future Directions

Significance:
This work establishes a critical pathway toward transparent AI. By converting opaque mathematical activations into natural language, it enables researchers and practitioners to debug models, audit for biases, and understand failure modes without needing deep expertise in linear algebra or neural architecture.

Future Directions Identified:

1. From Components to Circuits: Scaling description methods from individual units to entire computational subgraphs (circuits) to explain complex task execution.
2. Domain Specialization: Applying these methods to specialized domains (legal, medical, code) and multimodal/multilingual models.
3. Meta-Analysis of Interpreters: Critically analyzing the LLMs used as interpreters to understand their own biases and limitations.
4. Nuanced Polysemanticity: Developing methods that capture feature activation levels (intensity) rather than binary on/off states, acknowledging that a feature might represent different concepts at different activation strengths.
5. Rigorous Causal Benchmarks: The field urgently needs standardized benchmarks (similar to the MIB suite) with "gold" descriptions to systematically compare and improve description methods.

Conclusion:
The paper concludes that while concept descriptions are a powerful step toward demystifying LLMs, the field must move beyond correlational metrics toward rigorous, causal, and standardized evaluation to ensure these descriptions are truly faithful to the model's internal mechanisms.