Target-Oriented Pretraining Data Selection via Neuron-Activated Graph

1. Problem Statement

Large Language Models (LLMs) are increasingly used for specific downstream tasks (e.g., math reasoning, medical diagnosis). While pretraining on high-quality data is crucial, existing data selection methods suffer from two main limitations:

• Misalignment with Targets: General quality-based selection (e.g., filtering for educational content or high perplexity) often fails to align with specific downstream capabilities. High-quality general data does not necessarily trigger the specific internal mechanisms required for a target task.
• Black-Box Nature of Target-Oriented Methods: Prior approaches that attempt to align data with targets often rely on "black-box" signals, such as embedding similarity (e.g., BETR) or distilling performance-correlated losses into auxiliary classifiers. These methods lack interpretability, making it difficult to diagnose why certain data is selected or to refine the selection process.

The core challenge is to select pretraining data that specifically activates the internal computational pathways necessary for a target capability, without requiring additional training or relying on opaque representations.

2. Methodology: Neuron-Activated Graph (NAG)

The authors propose NAG-based Ranking, a training-free, interpretable framework that selects data based on the specific neurons it activates within an off-the-shelf LLM.

A. Core Concept

Instead of using sentence embeddings, the method characterizes a text input by the sparse set of high-impact neurons it activates in a Transformer-based LLM. The hypothesis is that inputs requiring similar capabilities will activate similar "functional backbones" of neurons across the model layers.

B. Step-by-Step Pipeline

1. Neuron Impact Quantification:

   • For a given input $h_{in}$ and a projection layer with weight matrix $W$, the impact of a specific neuron (column $W_{:,k}$) is defined as the magnitude of its contribution to the layer output:
     $$\text{Imp}(N_k | h_{in}) = |h_{in}^T W_{:,k}|$$
   • This measures how much the output changes if that specific neuron is deactivated.

2. Constructing the Neuron-Activated Graph (NAG):

   • For each input, the top-$K$ neurons with the highest impact scores are selected for each layer.
   • The NAG is defined as the collection of these layer-wise neuron indices:
     $$\text{NAG}(c) = \{ (\ell, k) \mid \ell \in \text{Layers}, k \in \text{Top-}K \text{ neurons in layer } \ell \}$$
   • This creates a compact, structured representation of the input's computational path.

3. Target-Oriented Ranking:

   • Target Profile: A small set of target examples ($D_{target}$) is used to aggregate a "Target NAG Profile." This profile calculates the frequency of each neuron being activated across the target set.
   • Similarity Calculation: For every candidate document in the pool ($D_{pool}$), its NAG is compared to the Target Profile. The similarity score $s(c)$ is computed as the weighted overlap between the candidate's activated neurons and the target profile's activation frequencies.
   • Selection: Candidates are ranked by $s(c)$, and the top fraction is selected for pretraining.

C. Key Design Choices

• Training-Free: Requires no fine-tuning or training of auxiliary classifiers.
• Layer Aggregation: Unlike methods using only final-layer embeddings, NAG aggregates signals across all layers, capturing distributed task-relevant information.
• Sparsity: It focuses on a very sparse subset of neurons (e.g., top 0.3% per layer), effectively isolating the "functional backbone."

3. Key Contributions

1. Novel Framework: Introduction of NAG-based Ranking, the first method to use sparse, layer-wise neuron activation patterns for target-oriented data selection.
2. Interpretability: The method provides a transparent mechanism for selection. It identifies which neurons are critical for a task, allowing for analysis of the model's "functional backbone."
3. Training-Free Efficiency: Eliminates the need for expensive classifier training or iterative distillation required by previous target-oriented methods.
4. Comprehensive Analysis: The paper provides deep insights into why NAG works, demonstrating that it captures a sparse set of critical neurons that are highly sensitive to specific tasks.

4. Experimental Results

The method was evaluated across six benchmarks (ARC-C, HellaSwag, TriviaQA, MMLU, XStoryCloze, XWinograd) using various backbone models (Qwen3, Llama-3.2, SmolLM3).

• Performance Gains:
  • vs. Random Sampling: NAG-based Ranking improved target-oriented pretraining by an average of 4.9% across benchmarks.
  • vs. State-of-the-Art: It outperformed the strong baseline BETR (embedding-based) by 5.3% on HellaSwag and 1.1%–4.1% in multi-target settings.
  • vs. Quality Heuristics: It outperformed FineWeb-Edu (a general quality classifier) by 2.4% on average, showing that general quality does not guarantee target capability.
• Multi-Target Robustness: In a multi-target setting (mixing goals for 6 benchmarks), NAG maintained gains (+3.6% over random), whereas BETR degraded significantly (-4.4%), indicating NAG's ability to handle diverse objectives without signal conflict.
• Model Agnosticism: The method yielded consistent improvements (+4.7% to +5.0%) regardless of the backbone model used for extraction (Qwen, Llama, SmolLM).

5. Significance and Analysis

The paper goes beyond empirical results to explain the mechanism of NAG:

• The "Functional Backbone": Ablation studies revealed that deactivating only 0.12% of neurons (those selected by NAG) caused a massive 23.5% performance collapse. In contrast, deactivating random neurons had negligible effects. This proves NAG isolates the critical, sparse neurons responsible for specific capabilities.
• Layer Importance: Restricting NAG to only the final layer caused a 4.1% average drop in performance. This confirms that task-relevant signals are distributed across the entire depth of the model, not just the output layer.
• Task Discriminability: Clustering analysis showed that NAG representations naturally group data by task type (e.g., math tasks cluster together, distinct from linguistic tasks), validating that NAG captures task-discriminative features.
• Efficiency: The selection pipeline is computationally cheap (one forward pass per document) and yields high "compute multipliers" (up to 3.7x efficiency gain on XStoryCloze), meaning models trained on NAG-selected data achieve the same accuracy with significantly fewer training tokens.

Conclusion

This paper establishes that neuron-level activation patterns are a superior signal for selecting pretraining data compared to embeddings or general quality heuristics. By constructing a Neuron-Activated Graph, the authors provide a scalable, interpretable, and highly effective method to steer LLM pretraining toward specific target capabilities, bridging the gap between data selection and the internal mechanisms of capability learning.