Technical Summary: Neuron Populations Exhibit Divergent Selectivity with Scale

Problem Statement

While neural scaling laws have established predictable relationships between model size and macroscopic observables (e.g., loss), the internal organization of representations at the neuron level remains poorly understood. A central question in deep learning and neuroscience is how neurons encode structure: whether representations are localized in single units or distributed across populations, and how these structures evolve as models scale. Specifically, it is unclear if cross-model recurring neuron populations follow systematic laws regarding their size, selectivity (monosemanticity vs. polysemanticity), and specialization.

Methodology

The authors investigate these questions using Rosetta Neurons, a class of neurons defined by Dravid et al. (2023) as units whose activation patterns recur across independently trained models. The study employs the following methodology:

1. Identification of Rosetta Neurons:

   • The authors analyze Transformer-based language models (Pythia, GPT-2, OPT, Qwen2.5; 80M–30B parameters) and vision models (OpenCLIP, DINOv2, Pixio, Diffusion models; 80M–5B parameters).
   • They align token-wise activations across models (using shared byte boundaries for text and resampled spatial grids for images).
   • They compute pairwise Pearson correlations between neurons across models and identify mutual nearest-neighbor pairs ($k=1$) as Rosetta Neurons. This filters out asymmetric or noisy matches.

2. Scaling Analysis:

   • They plot the number of discovered Rosetta Neurons against the total number of neurons in the model to derive scaling laws.
   • They compare these trends against null baselines, including untrained random networks and input-permuted datasets, to ensure the scaling is a property of learned representations rather than architectural artifacts.

3. Analytical Modeling:

   • The authors propose a capacity-allocation model based on the theory of superposition (Elhage et al., 2022).
   • The model assumes a power-law feature-importance spectrum ($w_r \propto r^{-\beta}$) where features compete for a finite number of neuron coordinates.
   • It posits that high-importance features are allocated "clean" isolation (monosemanticity) to reduce loss, while lower-importance features remain in superposition (polysemanticity).
   • The model derives that the number of detectable, isolated features (Rosetta Neurons) should scale as a sublinear power law ($N^{1/\beta}$) and predicts a Neuron Polarization Effect.

4. Property Evaluation:

   • Selectivity: Measured via vocabulary-space projection kurtosis for language models and VLM-as-a-judge evaluations for vision models.
   • Specialization: Analyzed by categorizing the top-activating contexts of Rosetta Neurons (e.g., code, math, general prose) across different model scales.
   • Utility: A case study filters data for continued pretraining using a single Rosetta Neuron to recover a specific domain (JavaScript).

Key Results

1. Sublinear Power-Law Scaling

Across both language and vision models, the population of Rosetta Neurons grows predictably with model size but follows a sublinear power law ($|R| \propto N^\alpha$ with $\alpha \approx 0.5\text{--}0.7$).

• Implication: While larger models contain more shared units in absolute numbers, these units occupy a shrinking fraction of the total neuron population.
• Validation: This trend is absent in untrained random networks and input-permuted baselines, confirming it arises from learned representations.

2. The Neuron Polarization Effect

The study observes a divergence in neuron properties as scale increases:

• Rosetta Neurons: Become increasingly selective and monosemantic. In language models, their vocabulary-space projections show increasing excess kurtosis; in vision models, VLM judges identify them as responding to coherent visual features more frequently.
• Non-Rosetta Neurons: Remain less selective and increasingly polysemantic, forming a crowded background of mixed features.
• Theoretical Alignment: This matches the analytical model's prediction where the "Rosetta frontier" moves outward, isolating high-importance features more cleanly, while the tail of lower-importance features remains superposed.

3. Domain Specialization

Rosetta Neurons shift toward specialized domains with scale.

• In language models, the firing of Rosetta Neurons becomes increasingly concentrated on specialized text categories (e.g., code, mathematics) relative to their frequency in the training corpus.
• This shift is not observed in non-Rosetta neurons, which maintain stable document-type preferences across scales.

4. Functional Utility in Data Filtering

A single Rosetta Neuron selective for JavaScript (identified in a Pythia-6.9B model) was used to filter a multilingual code corpus.

• Recovery: The filter achieved an F1 score of 0.98 in recovering the ground-truth JavaScript subset, nearly matching an oracle.
• Downstream Performance: Continued pretraining on the filtered data reduced perplexity to 3.02, nearly matching the oracle baseline (3.01) and significantly outperforming random filtering (3.59) or non-Rosetta neuron filtering (3.23).

Significance and Claims

The paper claims to establish a scaling law for interpretable, shared neuron-level structure. Its primary contributions are:

1. Linking Scale to Internal Structure: It demonstrates that model size systematically changes the universality, selectivity, and specialization of shared neuron populations, extending scaling laws beyond macroscopic loss metrics.
2. Polarization Mechanism: It provides empirical and theoretical evidence for a "Neuron Polarization Effect," where scaling drives a separation between a growing, highly selective population of shared units and a polysemantic background.
3. Practical Interpretability: It shows that Rosetta Neurons are not just theoretical curiosities but functional units that can be used for targeted tasks like data filtering and domain specialization without auxiliary training or target concepts.
4. Generalizability: The findings hold across heterogeneous architectures (Transformers, Diffusion Transformers) and modalities (language, vision), suggesting a fundamental principle in how neural networks organize representations under capacity constraints.

The authors conclude that Rosetta Neurons serve as a tractable observable for studying shared internal structure, offering a bridge between macroscopic scaling behavior and the microscopic organization of learned representations.