💬 NLP

FineScope : SAE-guided Data Selection Enables Domain Specific LLM Pruning and Finetuning

FineScope is a framework that leverages Sparse Autoencoders to curate domain-specific datasets for guiding structured pruning and self-data distillation, enabling the creation of compact, high-performing LLMs that outperform larger state-of-the-art models on specialized tasks.

Original authors: Chaitali Bhattacharyya, Hyunsei Lee, Junyoung Lee, Shinhyoung Jang, Il hong Suh, Yeseong Kim

Published 2026-03-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chaitali Bhattacharyya, Hyunsei Lee, Junyoung Lee, Shinhyoung Jang, Il hong Suh, Yeseong Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, all-knowing library (a Large Language Model or LLM) that contains every book ever written. It's incredibly smart and can talk about anything from quantum physics to 18th-century poetry. However, this library is huge, heavy, and expensive to run.

Now, imagine you only need a specialized pocket guide for a specific job, like fixing car engines or writing legal contracts. You don't need the whole library; you just need the chapters about cars or law. But here's the problem: if you try to shrink the library down to just those pages, you often lose the ability to understand the context, and the resulting "pocket guide" becomes clumsy and makes mistakes.

FineScope is a new, clever method to solve this problem. It's like a smart librarian who helps you shrink the library without losing the important stuff. Here is how it works, step-by-step:

1. The "Seed" and the "Magic Lens" (Data Selection)

Usually, to teach a model a specific skill, you need thousands of examples. But FineScope starts with just a few "seed" examples (like 10 sample questions about cars).

Instead of just looking at the words on the page, FineScope uses a Magic Lens called a Sparse Autoencoder (SAE).

The Analogy: Imagine the library's brain is a giant orchestra. When it reads a sentence about "cars," certain specific instruments (neurons) play loudly, while others stay quiet. The Magic Lens listens to which instruments are playing rather than just reading the sheet music.
The Result: The Lens scans a massive, messy pile of unlabeled books and finds the ones where the same instruments are playing. It filters out the noise and builds a perfect, tiny, high-quality dataset that is perfectly aligned with your specific topic.

2. The "Surgical Trim" (Pruning)

Now that we have this perfect, tiny dataset, we need to shrink the giant library.

The Old Way: People used to just cut out random pages or sections of the library to make it smaller. This often removed the very pages needed for the specific job, leaving the model confused.
The FineScope Way: We use our perfect dataset to guide a surgical trim. We ask the model: "Which parts of your brain are actually working hard when you read these car examples?"
The Result: We keep the "muscles" (neural pathways) that are strong and active for cars, and we gently remove the "muscles" used for cooking or history. The model becomes 35% smaller but stays super strong at its specific job.

3. The "Mentor" (Fine-Tuning & Distillation)

After the surgery, the model might feel a bit weak or forgetful.

The Analogy: Imagine a student who just had surgery. They need a mentor (the original, giant, smart model) to help them recover.
The Process: The mentor reads the perfect dataset and says, "Here is how a smart model would answer this." The smaller, trimmed model listens and tries to mimic the mentor's answers.
The Result: The small model learns to be just as good as the big one, but it's now lightweight and fast.

Why is this a Big Deal?

Efficiency: You can run these specialized models on cheaper computers or even phones, saving money and energy.
Quality: Even though the model is smaller, it actually performs better on specific tasks (like math or coding) than other methods because it wasn't forced to learn irrelevant things.
Simplicity: You don't need a massive team of experts to curate data. You just give the system a few examples, and it does the rest.

In short: FineScope is like taking a giant, general-purpose Swiss Army Knife, identifying exactly which tools you need for your specific job, and then re-forging a custom, lightweight tool that is lighter, faster, and sharper for that one task, without losing any of its cutting power.

1. Problem Statement

Large Language Models (LLMs) are typically trained on massive, diverse datasets to achieve broad generalization. However, real-world applications often require efficient, specialized models for narrow, domain-specific tasks (e.g., medical diagnosis, legal reasoning, or STEM education).

The Challenge: Deploying full-scale LLMs for these tasks is computationally expensive and often unnecessary. While model compression techniques like pruning exist, they typically rely on general-purpose data or random selection for fine-tuning.
The Bottleneck: Effective domain adaptation requires high-quality, domain-specific data. In practice, such datasets are rarely available, expensive to curate manually, or misaligned with the target domain. Consequently, pruning a model without domain-aligned data leads to severe performance degradation, as the model loses critical substructures needed for the specific task.

2. Methodology: The FineScope Framework

FineScope is a unified framework that tightly couples domain-aware data selection with structured pruning and fine-tuning. It operates in two main stages:

Stage 1: SAE-Guided Data Selection

Instead of relying on surface-level text similarity, FineScope uses the internal representations of the LLM to identify semantically relevant data.

Sparse Autoencoder (SAE) Training:
- The framework trains SAEs on intermediate activations of a pre-trained LLM (specifically, top- $K$ activation coordinates selected via Jacobian sensitivity).
- The SAE learns to compress these activations into a sparse latent space, effectively capturing the model's internal feature dictionary.
Seed-Based Retrieval:
- The user provides a small set of "seed" examples (e.g., ~10 samples) representing the target domain.
- The trained SAE encodes both the seed examples and a large, unlabeled corpus (e.g., OpenInstruct).
- Selection: The system computes the cosine similarity between the seed embeddings and the corpus embeddings in the SAE latent space. It selects the top- $M$ samples that are most semantically aligned with the seeds.
- Result: A compact, high-quality, domain-specific dataset ( $D_s$ ) is curated automatically.

Stage 2: Domain-Conditioned Pruning & Fine-Tuning

The curated dataset $D_s$ is used to guide the compression and recovery process.

Structured Pruning:
- Using LLM-Pruner, the framework performs structured pruning (removing entire blocks/heads).
- Key Innovation: The importance scores for pruning are calculated based on gradients derived from the domain-specific dataset $D_s$ , not a general corpus. This ensures that components critical for the target domain are preserved, while redundant general-purpose components are removed.
Teacher-Guided Distillation (TGD) Fine-Tuning:
- To recover knowledge lost during pruning, the pruned model is fine-tuned using a modified Self-Distillation approach.
- A "teacher" (the original unpruned model or a strong pre-trained model) generates outputs on the curated dataset. The pruned "student" model is trained to mimic these outputs, effectively transferring domain-specific behaviors back into the compressed architecture.

3. Key Contributions

Unified Framework: FineScope is the first framework to jointly leverage latent model representations (via SAEs) for both dataset construction and structured pruning, creating a closed loop where data selection informs model compression.
SAE for Retrieval: It introduces a novel method of using SAEs trained on intermediate activations to extract embeddings for retrieval. This captures deep semantic alignment with the model's internal logic rather than superficial token overlap.
Jacobian Sensitivity Selection: The paper proposes using Jacobian sensitivity (input sensitivity) to select the top- $K$ activation coordinates for SAE training, proving this is superior to random or magnitude-based selection for capturing task-relevant signals.
Domain-Conditioned Pruning: It demonstrates that pruning guided by domain-specific data preserves "domain-critical subnetworks," enabling aggressive compression without catastrophic forgetting.

4. Experimental Results

The authors evaluated FineScope on Vicuna-7B, MathCoder-CL-7B, and LLaMa 3.1-8B across STEM, Social Sciences, Humanities, Math, and Coding domains.

Performance Gains:
- FineScope consistently outperformed baselines (including random data selection, full-domain fine-tuning, and Alpaca-based tuning).
- On Math reasoning tasks, pruned models fine-tuned with FineScope achieved an average improvement of 11.50 points over baselines.
- In STEM, Social Sciences, and Humanities, FineScope-tuned pruned models often matched or exceeded the performance of much larger models (e.g., GPT-3 175B) despite having ~30% fewer parameters.
Pruning Resilience:
- Standard pruning with general data caused accuracy drops of up to 50%.
- FineScope enabled up to 35% parameter pruning while maintaining or improving accuracy.
- On the HumanEval coding benchmark, FineScope improved scores significantly (e.g., +0.27 for LLaMa 3.1) compared to tuning with full general-purpose corpora.
Data Efficiency: FineScope achieved superior results using only ~2,000 curated samples (vs. thousands of general instructions), proving that data quality and alignment outweigh quantity.
Ablation Studies:
- SAE Embeddings: Outperformed BERT and Sentence-Transformer embeddings.
- Jacobian Selection: Outperformed random, magnitude, variance, and PCA-based coordinate selection.
- Seed Sensitivity: The method is robust to the number of initial seeds (5–25), requiring minimal user input.

5. Significance

FineScope addresses a critical bottleneck in the deployment of LLMs: the trade-off between efficiency and specialization.

Resource Efficiency: It enables the creation of lightweight, domain-specialized models that can run in resource-constrained environments (e.g., edge devices) without sacrificing task performance.
Scalability: By automating data curation using SAEs, it removes the dependency on expensive, manually curated domain datasets.
Practical Impact: The framework provides a practical pipeline for organizations to compress large models for specific verticals (Law, Medicine, STEM) while retaining high accuracy, making specialized AI more accessible and cost-effective.

In summary, FineScope demonstrates that targeted data selection is the key to effective model compression, transforming the pruning process from a destructive operation into a specialized adaptation strategy.