Neurons Speak in Ranges: Breaking Free from Discrete Neuronal Attribution

1. Problem Statement

The paper addresses a fundamental limitation in the interpretability of Large Language Models (LLMs): neuron polysemanticity.

• The Issue: Traditional neuron interpretation methods assume a discrete, one-to-one mapping where a single neuron encodes a single semantic concept. However, neurons in LLMs are often polysemantic, meaning a single neuron activates for multiple, seemingly unrelated concepts.
• The Consequence: Current intervention techniques (e.g., masking or ablating a specific neuron to remove a concept) are "blunt instruments." Because a neuron encodes multiple concepts, disabling the entire neuron to suppress one concept inevitably degrades the model's performance on other concepts encoded by that same neuron, leading to collateral damage and suboptimal model control.
• The Gap: There is a lack of fine-grained methods to distinguish which part of a neuron's activation spectrum corresponds to a specific concept versus others.

2. Core Observation & Hypothesis

Through systematic analysis of both encoder (BERT, DistilBERT) and decoder (GPT-2, Llama-3) models across diverse datasets, the authors discovered a consistent statistical pattern:

• Gaussian-Like Distributions: For a given salient neuron, the activation magnitudes conditioned on a specific semantic concept form a distinct, approximately Gaussian (normal) distribution.
• Limited Overlap: While a single neuron may respond to multiple concepts (polysemanticity), the activation distributions for different concepts within that same neuron exhibit minimal overlap. They occupy distinct "bands" or ranges within the neuron's activation spectrum.
• Hypothesis: Instead of attributing a concept to an entire neuron, concepts should be attributed to specific activation ranges within that neuron. Intervening only on these specific ranges allows for precise manipulation of a target concept while preserving the neuron's function for other concepts.

3. Methodology: NeuronLens

The authors introduce NeuronLens, a framework for range-based attribution and intervention.

A. Range-Based Attribution

1. Data Collection: The model processes training samples labeled with specific concepts ($c$).
2. Statistical Modeling: For a selected salient neuron $j$ at layer $l$, the authors calculate the mean ($\mu$) and standard deviation ($\sigma$) of the activation values specifically for samples belonging to concept $c$.
3. Defining the Range: An attribution range ($AR$) is defined as:
   $$AR(l, j, c) = [\mu - \tau \times \sigma, \mu + \tau \times \sigma]$$
   Where $\tau$ is a hyperparameter (set to 2.5, covering ~98.76% of the distribution). This range captures the majority of concept-specific activations.

B. Causal Intervention (Range-Based Masking)

Instead of zeroing out the entire neuron output, NeuronLens applies a range-gated operator ($\phi$):

• If the neuron's activation $h_j(x)$ falls within the calculated $AR$, the intervention is applied (e.g., zeroing out the value).
• If the activation falls outside the range, the value is passed through unchanged.
• Formula:
  $$h'_j(x) = \begin{cases} \phi(h_j(x)) & \text{if } h_j(x) \in AR(l, j, c) \\ h_j(x) & \text{otherwise} \end{cases}$$

C. Alternative Interventions

The paper also explores and proposes Adaptive Dampening, where the suppression strength is proportional to the distance of the activation from the mean, further reducing disruption to the model's natural dynamics.

4. Key Contributions

1. Empirical Discovery of Distributional Structure: Demonstrated that concept-conditioned neuron activations follow Gaussian-like distributions with distinct means and limited overlap, even in highly polysemantic neurons.
2. NeuronLens Framework: Proposed a novel method to map concepts to activation ranges rather than discrete neurons, enabling fine-grained interpretability.
3. Causal Validation: Showed that range-based interventions successfully erase target concepts while significantly preserving auxiliary concepts and general model capabilities compared to full neuron masking.
4. Complementarity with SAEs: Demonstrated that NeuronLens can be applied to Sparse Autoencoder (SAE) features, suggesting it can further disentangle representations learned by SAEs.

5. Experimental Results

The authors evaluated NeuronLens on models like BERT, GPT-2, and Llama-3.2-3B across datasets including AG-News, Emotions, DBPedia-14, IMDB, and SST2.

• Target Concept Suppression: Range-based masking achieves suppression of the target concept comparable to full neuron masking (e.g., $\Delta$Acc drops of ~0.90 for Llama on DBPedia).
• Reduced Collateral Damage:
  • Auxiliary Concepts: Range-based masking causes significantly less degradation to non-target concepts. For example, on Llama-3, full neuron masking caused a ~0.91 drop in auxiliary accuracy, whereas range masking caused only a ~0.66 drop (a reduction in collateral damage of ~25 percentage points in some settings).
  • General Capabilities: Full neuron masking severely degrades general language modeling (high Perplexity increase) and reasoning (MMLU score drop). Range-based masking maintains MMLU performance (often improving or staying stable) and incurs minimal Perplexity increases (e.g., +0.5 to +0.8 vs. +7 to +12 for full masking).
• Robustness: The advantage of range-based masking becomes more pronounced as the percentage of masked neurons increases, avoiding the "performance cliff" seen in discrete masking.

6. Significance and Impact

• Paradigm Shift: The paper challenges the "discrete neuron" paradigm, arguing that interpretability and control must operate at the level of activation distributions.
• Model Safety & Control: Provides a more precise tool for "concept erasure" (e.g., removing toxic or biased concepts) without breaking the model's general utility, a critical requirement for deploying safe LLMs.
• Efficiency: Unlike methods requiring retraining or training new bases (like SAEs), NeuronLens operates on existing model representations with minimal computational overhead.
• Future Directions: Suggests that future interpretability tools should focus on the continuous spectrum of activation rather than binary neuron states, potentially integrating with representation engineering and steering techniques.

In summary, NeuronLens proves that while neurons are polysemantic, the ways they express different concepts are statistically separable. By targeting these specific activation ranges, researchers can surgically edit LLM behavior with unprecedented precision.