💬 NLP

Select or Project? Evaluating Lower-dimensional Vectors for LLM Training Data Explanations

This paper demonstrates that greedily selecting a small, architecturally informed subset of model components is both more effective and computationally efficient than using full gradients or random projections for generating instance-based explanations of large language models.

Original authors: Lukas Hinterleitner, Loris Schoenegger, Benjamin Roth

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lukas Hinterleitner, Loris Schoenegger, Benjamin Roth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand why a giant, super-smart robot made a specific decision. You know the robot was trained on a massive library of books, but which specific book taught it that particular fact? This is the world of "instance-based explanations" in artificial intelligence. Scientists want to trace a robot's answer back to the exact training example that influenced it, almost like a detective looking for a fingerprint. To do this, they usually look at the robot's "gradients." Think of a gradient as a giant, multi-dimensional map showing exactly how the robot's brain would change if it saw a specific sentence. The problem is, for modern robots (called Large Language Models), these maps are so huge they are impossible to carry around. A single map for a standard model takes up over 4 gigabytes of memory—more than a whole movie file! Because of this, researchers have to shrink these maps to make them manageable. They've been trying two main tricks: either throwing away most of the map and keeping only a few random pieces, or squashing the whole map down into a tiny, blurry summary. But nobody really knew which trick was actually better at helping the detective find the right book.

This paper, titled "Select or Project?", sets out to solve that mystery. The authors, Lukas Hinterleitner and his team from the University of Vienna, built a new testing ground to see if it's better to carefully pick a few important parts of the robot's brain (Selection) or to mathematically squish the whole brain into a smaller space (Projection). They used a clever game to test this: they took a sentence, rephrased it in a different way, and asked the robot's "gradients" to find the original sentence among a crowd of look-alikes. If the gradients are good, they should point straight to the original.

The results were surprising. The team found that the "squishing" method (Random Projection), which tries to keep the shape of the whole map, wasn't the best detective. Instead, a method they called "Greedy Selection" won the game. This approach is like a smart librarian who doesn't read the whole library but knows exactly which three shelves hold the most important clues. By carefully picking a small, specific set of the robot's internal parts (specifically, certain layers that handle logic and word connections), they created a tiny, super-sharp map. In their tests, this targeted selection was not only more accurate at finding the right training example but also much faster and cheaper to compute.

Here is the kicker: the full, giant map of the robot's brain actually performed worse than this tiny, carefully chosen subset. In one difficult test where the robot had to generate a new story based on a rephrased prompt, the full map only got the answer right about 22% of the time—barely better than guessing. However, the small, selected group of parts got it right about 36% of the time. The authors suggest that the full map is actually "noisy," filled with so much extra information that it drowns out the real clues. By selecting only the most informative pieces, they cut out the noise and got a clearer signal.

The paper also looked at which parts of the robot's brain were the best detectives. They found that it wasn't about the size of the part; a tiny piece of the brain could be more useful than a huge one. Instead, it was about what the part did. The parts responsible for the robot's internal logic (called MLP layers) were much better at finding the right training data than the parts responsible for paying attention to specific words (like the "Key" and "Value" parts of the attention mechanism). Interestingly, the parts at the very beginning and very end of the robot's processing chain were the most helpful, while the middle parts seemed to get confused and less useful.

In the end, the authors conclude that for explaining how these giant AI models work, we don't need the whole picture. We just need the right few pieces. Their "Greedy Selection" strategy is a practical, efficient way to make these explanations possible without needing supercomputers that run for hundreds of hours. While they tested this on a specific model with about 1.2 billion parameters, their findings suggest that for the future of AI transparency, being picky and selective is far better than trying to keep everything.

Technical Summary: Select or Project? Evaluating Lower-dimensional Vectors for LLM Training Data Explanations

Problem Statement
Gradient-based methods for instance-based explanations of Large Language Models (LLMs) face a significant computational barrier due to the immense dimensionality of model gradients. For a 1-billion parameter model, a single gradient vector requires over 4 GB of memory, rendering full-gradient analysis intractable. Current practices often restrict influence estimation to a subset of model parameters (e.g., specific layers or LoRA adapters) to make computation feasible. However, these subsets are frequently chosen ad hoc without systematic justification. Furthermore, alternative dimensionality reduction techniques, such as random projection, are used but lack comparative evaluation against architecture-aware selection strategies. The core problem is determining whether it is more effective to select a sparse, structurally informed subset of model components or to project the full high-dimensional gradient into a lower-dimensional space to capture training data influence.

Methodology
The authors propose a novel, re-training-free benchmark to evaluate dimensionality reduction strategies for instance-based explanations. The evaluation frames the problem as a retrieval task: given a test instance (specifically, a paraphrased version of a training sample or a model-generated completion), the system must identify the original training example among a set of distractors based on gradient similarity.

Gradient Representation: The study utilizes loss gradients with respect to model parameters ( $\nabla_\theta L$ ). The full gradient is constructed by flattening component gradients (e.g., query, key, value projections, MLP gates) into a single vector.
Evaluation Setup:
- Datasets: The authors use the LIMA subset of the Tülu 2 dataset. They generate two query sets: a Paraphrased set ( $D_p$ ) using GPT-4o-mini to rephrase prompts while preserving semantics, and a Model-Generated set ( $D_m$ ) where the fine-tuned model generates completions for paraphrased prompts.
- Metric: Retrieval accuracy is measured by checking if the cosine similarity between the query gradient and the original training sample's gradient is higher than its similarity to distractors (selected via BM25).
Strategies Compared:
- Full Gradient: Using the entire parameter set (baseline).
- Random Projection (Project): Projecting the full gradient into a lower-dimensional space ( $d \ll M$ ) using a random matrix, implemented component-wise to avoid materializing massive matrices. This preserves the geometric structure of the full gradient.
- Greedy Component Selection (Select): An architecture-aware algorithm that iteratively selects a subset of model components (up to a parameter budget) that maximizes retrieval accuracy. This leverages the linearity of dot products to reconstruct similarity scores from pre-computed scalar values without storing high-dimensional vectors.

Key Contributions

Novel Benchmark: The introduction of an efficient, retrieval-based benchmark designed to test how well different low-dimensional representations support the identification of influential training examples.
Principled Comparison: A systematic evaluation comparing architecture-aware greedy selection against geometry-preserving random projection and full-gradient baselines.
Efficiency and Accuracy Demonstration: Empirical evidence showing that targeted component selection yields more accurate attributions and is significantly more computationally efficient than random projection.

Results

Full Gradient Limitations: While the full gradient performs well in the paraphrased setting (0.993 accuracy), its performance collapses in the more challenging model-generated setting (0.218 accuracy), barely outperforming random chance (0.20).
Component Variance: Not all components contribute equally. In the model-generated setting, MLP gate and up projections carry the most informative signal, whereas attention key and value projections contribute little or perform worse than random. Parameter count does not correlate linearly with performance; functional role is a more critical determinant.
Selection vs. Projection:
- Accuracy: A greedily selected subset of components (using less than 5% of parameters) achieves near-perfect accuracy (0.998) in the paraphrased setting and significantly outperforms the full gradient in the model-generated setting (0.36 vs. 0.218).
- Geometry vs. Utility: While random projection effectively preserves the global geometry of the full gradient (high cosine similarity to full gradient scores), this high fidelity does not translate to optimal retrieval accuracy. Selecting components based on their specific utility for the retrieval task is superior to preserving global geometry.
Computational Efficiency: The greedy selection method is substantially more efficient. Pre-computing dot products took ~4 hours, with selection taking minutes. In contrast, random projection baselines required over 900 hours (without caching) due to the computational intensity of projecting massive vectors.

Significance and Claims
The paper claims that for instance-based explanations of large models, targeted component selection is a practical and superior strategy compared to random projection or using the full gradient. The central insight is that a carefully chosen, sparse subset of model components can provide a clearer and more discriminative signal for specific tasks than the full, noisy gradient. This approach makes instance-based explanations for large models more computationally feasible without sacrificing, and often improving, attribution accuracy. The authors conclude that restricting influence estimation to a structurally informed subset of layers is a viable path forward, challenging the assumption that the full gradient is the optimal source of information.

The work is modest in its scope, noting limitations regarding the model size (1.2B parameters) and the static nature of the evaluation (using a single checkpoint). The authors suggest future work should explore generalization across different architectures, scales, and dynamic attribution methods that track influence throughout the training process.

Technical Summary: Select or Project? Evaluating Lower-dimensional Vectors for LLM Training Data Explanations

More like this