💬 NLP

Influcoder: Distilling Decoders' Gradient Influence Rankings into an Encoder for Data Attribution

The paper introduces Influcoder, a method that distills gradient influence rankings from decoders into an encoder to enable fast, cost-effective, and scalable data attribution for large language models.

Original authors: Dimitri Kachler, Damien Sileo, Pascal Denis

Published 2026-06-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dimitri Kachler, Damien Sileo, Pascal Denis

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, incredibly smart robot chef (a Large Language Model) who has learned to cook by reading millions of recipes and food blogs. Sometimes, this chef accidentally learns to make toxic or dangerous dishes because one specific, bad recipe was in the pile.

The Problem: Finding the Bad Recipe
You want to know: "Which specific recipe in the millions caused the chef to make that bad dish?" This is called Data Attribution.

Traditionally, to find this "culprit recipe," researchers use a method called Influence Functions. Think of this like trying to re-bake every single dish in the world, one by one, while removing one ingredient at a time to see how the taste changes. It's incredibly accurate, but it's also painfully slow and requires a massive kitchen (computer storage) to keep track of all the calculations. It's like trying to find a needle in a haystack by rebuilding the entire haystack every time you look.

The Solution: Influcoder
The paper introduces a new tool called Influcoder. Instead of doing the heavy lifting every time you need an answer, Influcoder is like a super-fast, trained detective.

Here is how it works, using a simple analogy:

The Training Phase (Stage 1):
First, the researchers take a small, manageable group of recipes (data) and the robot chef. They do the slow, heavy math to figure out exactly which recipes influence the chef's behavior the most. They write these answers down in a "Gold Standard Answer Key."
- The Trick: They don't just look at the words; they look at the "mathematical fingerprints" (gradients) of how the chef learned from each recipe.
The Distillation (Stage 2):
Next, they train a smaller, simpler model (the "Encoder" or the Detective) to look at a recipe and guess: "Based on the Gold Standard Answer Key, how much influence does this recipe have?"
- Think of this as teaching a junior detective to recognize patterns. The junior detective doesn't need to re-bake the dishes; they just need to look at the recipe and say, "Ah, this looks like the kind of thing that makes the chef angry," based on what they learned from the senior detective.
The Result:
Once trained, this "Detective" can scan millions of recipes in seconds. It doesn't need the giant kitchen anymore. It just looks at the recipe, checks its internal notes, and gives you a ranking of which ones are the most influential.

Why is this a big deal?

Speed: The paper claims Influcoder is 15 to 100 times faster than the current best methods. It's like switching from walking across the country to taking a bullet train.
Storage: It takes up way less space. Instead of needing a warehouse to store the math for every recipe, you just need a small notebook.
Effectiveness: In tests, the detective was almost as good at finding "toxic" (dangerous) recipes as the slow, heavy methods, but it did it in a fraction of the time.

The Catch (Limitations)
The paper is honest about the downsides. This "Detective" is trained specifically on the type of recipes it studied.

If you ask it to find bad recipes in a completely different language or a totally different style of cooking it hasn't seen before, it might get confused and perform poorly.
It needs a bit of setup time to learn the "Gold Standard" first, whereas older methods are "plug-and-play" (though they are slow).

In Summary
Influcoder is a method that teaches a small, fast AI model to mimic the expensive, slow calculations of a large AI. It allows us to quickly identify which specific pieces of data are responsible for a model's behavior (like toxic outputs), making it practical to clean up huge datasets without waiting weeks for the computer to finish the math.

Technical Summary: Influcoder

Problem Statement

As Large Language Models (LLMs) grow in capability, the need to curate high-quality training datasets has intensified. Specifically, there is a demand for Data Attribution (DA) methods that can identify which specific training samples precondition a model to generate certain outputs, such as toxic behavior. While Influence Functions (Koh and Liang, 2017) provide a theoretical framework for this by quantifying the effect of training samples on test loss, their practical application at scale is hindered by prohibitive computational costs and storage requirements. Existing methods often require per-example gradients or inverse Hessian approximations, which are unstable and slow. Furthermore, recent studies (Li et al., 2025; Ivison et al., 2025) indicate that many influence methods struggle to outperform random baselines when scaling up, while others (Jiao et al., 2025) show promise for filtering but remain computationally expensive.

Methodology: Influcoder

The authors propose Influcoder, a method that distills gradient influence rankings into a compact encoder model to achieve efficient, scalable data attribution. The approach avoids the instability of inverse Hessian estimation by relying on a first-order influence formulation based on cosine similarity of LoRA (Low-Rank Adaptation) gradients.

1. Gradient Influence Formulation

Instead of using the full inverse Hessian, the method adopts a first-order influence formulation (Garima et al., 2020) restricted to LoRA adapter parameters ( $\theta_L$ ). To ensure stability and avoid over-weighting high-norm gradients, the dot product is replaced with cosine similarity:
$\text{Inf}'(p, q) = \cos(\nabla\ell(\theta_L, p), \nabla\ell(\theta_L, q))$
where $p$ is a pool (training) sample and $q$ is a query sample.

2. Two-Stage Training Pipeline

The method operates in two distinct stages:

Stage 1: Ground Truth Label Collection
- Pool ( $P$ ) and query ( $Q$ ) samples are passed through a target decoder model.
- Gradients are computed only on the response tokens (masking the prompt), as these are deemed more informative.
- To reduce storage, gradients are projected to a lower dimension using CountSketch (Charikar et al., 2004).
- The "ground truth" influence matrix ( $I_{GT}$ ) is pre-computed by calculating the cosine similarity between all pairs of pool and query gradients.
Stage 2: Distillation into an Encoder
- An encoder model (e.g., Ettin-encoder) is trained to predict influence scores directly from input samples, bypassing the need to compute gradients for every query.
- The encoder produces embeddings for pool and query samples ( $E_p, E_q$ ).
- Predicted influences ( $I_P$ ) are calculated as the cosine similarity between these embeddings.

3. Ranking-Based Loss Function

Since the goal is to rank samples rather than predict exact influence values, the training objective combines two differentiable ranking losses:

Pearson Correlation Loss ( $L_{Pearson}$ ): Aligns the entire batch of predicted influences with the ground truth matrix to ensure global consistency.
KL Divergence Loss ( $L_{KL}$ ): A softmax-based loss applied per query to specifically optimize the ranking of the top-k most influential samples.
Total Loss: $L_{Total} = \alpha L_{Pearson} + (1 - \alpha) L_{KL}$ .

Key Contributions

Novel Distillation Approach: Influcoder is the first method to distill gradient influence rankings (specifically LoRA-based) into an encoder, distinguishing it from prior distillation works that focus on training losses (e.g., AirRep) or full gradient proxies (e.g., IProX).
Scalability and Efficiency: By replacing expensive gradient computations with a forward pass through a small encoder, the method achieves significant speedups and storage reductions.
Practical Filtering: The method demonstrates efficacy in filtering toxic data, a critical application for LLM safety.

Experimental Results

The authors evaluated Influcoder against baselines including LESS, LoGRA, DataInf, RDS+, and TF-IDF.

Inference Speed and Cost:
- Influcoder (68M parameters) processes samples 15x faster than the closest proxy model, 45x faster than full LoGRA, and 100x faster than LESS.
- In terms of FLOPs, Influcoder uses 5x fewer than the nearest proxy and 44x fewer than LESS.
- Storage requirements are drastically lower (30 MB for Influcoder vs. 320 MB for LESS and 2.24 GB for LoGRA).
Attribution Quality:
- Ranking Correlation: Influcoder achieves Mean-Aggregated Spearman correlations comparable to or slightly below LESS, significantly outperforming semantic similarity baselines and other projection-based methods.
- Toxicity Filtering: In the XSTest-response task (filtering toxic examples), Influcoder achieved an AUPRC of 0.6981 (Heterogeneous) and 0.6919 (Homogeneous). While slightly lower than LESS (0.7075 and 0.7562), it is vastly superior to DataInf (0.2157) and Semantic baselines. Crucially, Influcoder achieves this with an inference time of ~32 seconds compared to ~2248 seconds for LESS.

Significance and Limitations

The paper claims that Influcoder provides a viable path toward effective data attribution at large scales, balancing the trade-off between the high accuracy of gradient-based methods and the speed of representation-based methods. It enables the use of influence functions for tasks like toxicity filtering where speed is essential.

However, the authors explicitly note significant limitations:

Distribution Dependence: Unlike methods such as LoGRA or LESS which function independently of the data distribution, Influcoder's performance drops dramatically if samples fall outside the distribution of the pool and query sets used during training.
Training Data Requirement: The method requires a sufficient amount of training data to distill the influence initially. In tasks with insufficient data, performance degrades.
Setup Overhead: Unlike "plug-in" methods, Influcoder requires an initial setup phase involving gradient stocking and encoder training, which adds complexity to the workflow.

In conclusion, Influcoder offers a cost-effective, high-speed alternative for data attribution, particularly suitable for scenarios where the data distribution is known and rapid inference is prioritized over the absolute highest ranking precision.