EvoSelect: Data-Efficient LLM Evolution for Targeted Task Adaptation

1. Problem Statement

The paper addresses the challenge of efficiently adapting Large Language Models (LLMs) to specific target tasks using iterative synthetic data generation. While iterative generation–training loops (where an external generator creates data, the model is fine-tuned, and the process repeats) are popular, they suffer from two critical issues:

1. Task Misalignment: Generated samples tend to drift away from the target task distribution over iterations, leading to reduced adaptation efficacy.
2. Redundancy: Successive iterations often produce highly overlapping or redundant samples, wasting computational resources and diluting useful learning signals.

Existing data selection methods typically address these issues in isolation:

• Attribution-only methods (e.g., gradient similarity) select samples close to the target distribution but often cluster around a single centroid, failing to cover the full distribution and causing redundancy.
• Diversity-only methods ensure coverage but often select samples that are irrelevant to the specific target task.
• Heuristic combinations (e.g., weighted sums) fail to synergize these objectives effectively.

The core problem is how to select a subset of synthetic data in each iteration that simultaneously maximizes target task alignment and sample diversity without relying on expensive human labeling.

2. Methodology: EVOSELECT

The authors propose EVOSELECT, a framework that integrates data attribution and diversification into a unified, principled optimization loop. The method operates within an iterative generation–selection–training cycle.

Key Components:

1. Proxy Model & Gradient Representations:

   • To avoid the prohibitive cost of computing gradients for the full LLM, EVOSELECT uses a smaller proxy model to extract gradient features for both training candidates and validation (target) samples.
   • These gradients are compressed and normalized to form feature vectors $g(x)$.

2. Target Alignment via Optimal Transport (OT):

   • Instead of matching samples to a single centroid (as in standard attribution), EVOSELECT uses Optimal Transport to align the distribution of selected training samples with the full validation distribution.
   • It defines a cost matrix $C$ based on the pairwise distances between training and validation gradient features.
   • The objective minimizes the transport cost, ensuring the selected subset covers the entire geometry of the target distribution, not just the densest regions.

3. Diversity Regularization:

   • To prevent redundancy, a diversity term is introduced based on the similarity kernel between training samples ($S = \langle g_i, g_j \rangle$).
   • This acts as a penalty: if a sample is selected, its similar neighbors are discouraged from being selected, promoting coverage of complementary regions in the data space.

4. Unified Optimization (Algorithm 1):

   • EVOSELECT formulates a joint optimization problem. It iteratively updates the weights $w$ of the candidate samples.
   • The update step combines the OT gradient (pulling samples toward the target distribution) and the diversity gradient (pushing samples apart to reduce redundancy).
   • The weights are updated via an exponentiated gradient step and normalized to form a probability distribution.
   • Finally, the top-$k$ samples with the highest weights are selected for fine-tuning.

3. Key Contributions

• Novel Paradigm: Introduces an iterative generation–selection–training loop that explicitly models the trade-off between task alignment and diversity, moving beyond simple heuristic combinations.
• Optimal Transport for Alignment: Proposes using OT to align the entire training distribution with the target distribution, solving the "centroid bias" problem inherent in gradient-based attribution methods.
• Synergistic Framework: EVOSELECT is the first method to directly synergize attribution and diversity objectives within a single optimization objective for targeted LLM adaptation, rather than treating them sequentially.
• Efficiency: Utilizes a proxy model and compressed gradient representations to make the selection process computationally feasible for large-scale LLMs.

4. Experimental Results

The authors evaluated EVOSELECT on 10 knowledge-intensive benchmarks (covering science, commonsense, logical, and biomedical reasoning) using both strong (14B generator, 3B base) and weak (14B generator, 14B base) generator settings.

• Performance Superiority: EVOSELECT consistently outperformed all baselines (Random, Attribution-only, Diversity-only, Attr-Div, and TSDS) across different selection ratios (20% and 50%) and iterations.
• Stability: Unlike other methods which sometimes degraded the base model's performance (harmful adaptation), EVOSELECT was the only method that consistently improved upon the base model performance throughout the evolution process.
• Robustness to Generator Strength: The method remained effective even when the data generator was not strictly stronger than the base model, indicating it identifies fundamentally useful data regardless of generator capacity.
• Task Difficulty: The performance gain of EVOSELECT over baselines increased as the difficulty of the target task increased (i.e., when the base model's initial accuracy was lower).
• Comparison to Full Data: In many cases, EVOSELECT achieved performance comparable to or better than training on the entire generated dataset, despite using only a fraction (20-50%) of the data.

5. Significance

• Data Efficiency: EVOSELECT demonstrates that high-quality adaptation does not require massive amounts of synthetic data; rather, it requires principled selection to filter out noise and redundancy.
• Scalability: By leveraging proxy models and efficient OT solvers, the framework is scalable to large LLMs where full gradient computation is infeasible.
• Theoretical Insight: The paper highlights that naive selection strategies (like centroid matching) fail to capture the geometric structure of task distributions. The use of Optimal Transport provides a mathematically rigorous way to ensure both relevance and coverage.
• Practical Impact: This approach offers a viable solution for adapting LLMs to specialized domains (e.g., medicine, law) where high-quality human-labeled data is scarce, and synthetic data generation is the primary source of training material.