🤖 machine learning

On the Fragility of Data Attribution When Learning Is Distributed

This paper demonstrates that data attribution in distributed learning is fragile, as a malicious participant can exploit latent optimization to inject synthetic batches that significantly inflate their measured contribution without degrading global model utility or triggering existing defenses.

Original authors: Xian Gao, Bo Hui, Min-Te Sun, Wei-Shinn Ku

Published 2026-05-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xian Gao, Bo Hui, Min-Te Sun, Wei-Shinn Ku

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Credit Card" Problem

Imagine a group of neighbors trying to build a giant, shared garden (the Machine Learning Model). Each neighbor brings a different set of seeds and tools (their Data). Some neighbors have rare, exotic flowers; others only have common weeds.

To keep everyone motivated, the group leader uses a special calculator called Data Attribution. This tool tries to figure out exactly how much each neighbor contributed to the garden's final beauty. Based on this score, neighbors get paid, get credit, or get to keep their spot in the club.

The paper's main discovery: A sneaky neighbor can trick this calculator. They can make it look like they brought the most valuable seeds in the whole group, even though they didn't actually help the garden grow any better. In fact, the garden looks exactly the same as if they had played fair.

The Setup: How the Attack Works

The researchers found a way for a single "bad actor" to game the system without getting caught. Here is how they did it, broken down into steps:

1. The "Ghost Seeds" (Synthetic Data)

Usually, if you want to cheat, you might bring fake, broken seeds (bad data) to ruin the garden. But that's obvious; the garden would look ugly, and you'd get kicked out.

Instead, this attacker uses Latent Optimization. Think of this as a "magic seed printer." The attacker has a blueprint (a decoder) that can print tiny, perfect-looking seeds. These aren't real seeds from their own land; they are generated by a computer.

2. The "Missing Puzzle Piece" Strategy

The garden is missing some specific types of flowers because the other neighbors didn't bring them. The attacker's magic printer creates just enough of these missing flowers to fill the gaps.

Why this matters: The credit calculator (the attribution tool) loves "completeness." It thinks, "Wow, this neighbor filled the holes in our garden! They must be super helpful!"
The trick: The attacker only prints just enough to look helpful, but not enough to mess up the garden's overall look.

3. The "Perfect Mimic" (Stealth)

To avoid getting caught, the attacker makes sure their contribution looks exactly like a normal, honest neighbor's contribution.

They match the size of the contribution (so it doesn't look too big).
They match the direction (so it pushes the garden in the same way everyone else does).
They make sure the final garden (the model's accuracy) looks just as beautiful as it would have without them.

The Result: The "Invisible" Heist

The paper ran this experiment with different types of gardens (datasets like CIFAR-10 and FashionMNIST) and different garden planners (models like ResNet and VGG).

What happened?

The Score: The sneaky neighbor's "contribution score" skyrocketed. They went from being at the bottom of the list to the top, or at least near the top.
The Garden: The garden's quality (accuracy) did not drop. It stayed exactly the same.
The Defenses: The group's security guards (defenses that look for weird shapes or broken plants) didn't notice anything wrong because the "ghost seeds" looked so normal.

The Analogy: The "Perfectly Polished" Lie

Imagine a team of chefs making a soup. The boss asks, "Who added the most flavor?"

Normal Chefs: Add real ingredients.
The Attacker: Instead of adding a huge, obvious pile of salt (which would ruin the soup), they add a tiny, invisible pinch of a "flavor enhancer" that the boss's taste-tester loves.
The Outcome: The soup tastes exactly the same as before (no one complains), but the taste-tester's machine gives the attacker a massive bonus because the machine thinks that tiny pinch was the secret ingredient that made the soup perfect.

Why This Matters (According to the Paper)

The paper warns us that trust is fragile.

We are starting to use these "contribution scores" to decide who gets paid for data, who owns the model, and how to govern AI systems.
The paper shows that these scores can be manipulated easily. A bad actor can steal credit without hurting the system's performance.
Current security measures (which check if the model is broken or if the data looks weird) do not work against this specific type of trick.

Summary

The paper proves that in a distributed learning system, you cannot trust the "scorecard" just because the final result looks good. A clever participant can use a "magic printer" to create fake-but-perfect data that tricks the scoring system into giving them a massive reward, all while leaving the actual product unchanged and undetectable.

The takeaway: If you are paying people based on how much they "contributed" to an AI, you need a new way to check their work, because the current scorecards can be fooled.

Technical Summary: On the Fragility of Data Attribution When Learning Is Distributed

Problem Statement
Data attribution has become a critical component for pricing, auditing, and governance in machine learning pipelines, particularly in distributed settings like Federated Learning (FL). These systems rely on attribution metrics (e.g., Shapley values, influence functions) to quantify the contribution of individual data owners to global model utility. However, current attribution methods implicitly assume that these scores faithfully reflect participants' actual contributions. This paper challenges that assumption, demonstrating that in standard distributed training workflows, a single malicious participant can strategically manipulate their measured attribution value to appear significantly more beneficial than they are, without degrading the global model's utility. The core vulnerability lies in the sensitivity of marginal-utility estimators to non-IID label coverage and the specific structural preferences of attribution evaluators.

Methodology: Latent Optimization Attack
The authors propose an "attribution-first" threat model where a malicious client ( $i^\star$ ) aims to maximize its attribution score $E(\hat{g}^\star_t)$ while strictly preserving global utility $U(\cdot)$ within a small tolerance $\delta$ . Unlike traditional poisoning attacks that degrade performance, this attack operates under the constraint that test accuracy remains stable.

The proposed Latent Optimization Attack proceeds as follows:

Synthetic Data Injection: The attacker utilizes a fixed, pre-trained decoder ( $Dec(\cdot)$ ) to generate a small batch of synthetic samples $(\tilde{x}, \tilde{y})$ from a latent vector $z$ .
Latent Vector Optimization: The attacker optimizes the latent vector $z$ $z$ via gradient descent to satisfy a joint loss function $L(z) = L_1(z) + L_2(z) + L_3(z)$ $L (z) = L_{1} (z) + L_{2} (z) + L_{3} (z)$ :
- $L_1$ (Direction Alignment): Ensures the gradient of the synthetic batch aligns with a benign reference direction (e.g., the global model update $w_t - w_{t-1}$ ) using cosine similarity.
- $L_2$ (Norm Consistency): Matches the magnitude of the synthetic gradient to that of benign updates to evade geometry-based defenses.
- $L_3$ (Task Fidelity): Maintains standard cross-entropy loss on the synthetic batch to ensure the data is class-discriminative and useful for training.
Strategic Coverage: The attacker specifically targets classes that are underrepresented or absent in their local real dataset ( $D^{(i^\star)}_r$ ). By injecting synthetic samples for these "missing" labels, the attacker exploits the evaluator's sensitivity to coverage gaps.
Hybrid Update: The attacker mixes the optimized synthetic batch with their real data to perform local training, generating a reported update $\hat{g}^\star_t$ that is statistically indistinguishable from benign updates but yields disproportionately high marginal utility in the eyes of the server-side evaluator.

Key Contributions

Attribution-First Threat Model: The paper formalizes a new attack objective in FL where the goal is to inflate data-attribution scores under explicit utility and plausibility constraints, distinct from traditional model-poisoning objectives.
Practical Latent-Optimization Attack: The authors introduce a concrete attack mechanism that uses decoder-generated synthetic data to construct attribution-aware local updates. This method remains utility-preserving and statistically plausible, evading standard sanity checks.
Theoretical and Empirical Validation: The paper provides theoretical insight into why such updates increase marginal attribution without degrading accuracy (due to alignment with global descent directions and coverage-driven utility gains). Empirically, the attack is validated across three datasets (CIFAR-10, FashionMNIST, SVHN), three model architectures (ResNet-18, WRN-28-10, VGG16 BN), and multiple evaluators (Federated Shapley Value and Leave-One-Out).

Experimental Results

Attribution Inflation: Under the Latent Optimization attack, a malicious client that initially ranked last (receiving near-zero attribution) consistently moved to the upper half of the ranking, often achieving attribution values comparable to or exceeding benign high-contributors. For instance, on CIFAR-10 with ResNet-18, the malicious client's attribution value increased from 0.0547 to 0.1682, shifting its rank from 10th to 2nd.
Utility Preservation: Unlike baseline attacks (Label Flipping, Random Noise) which significantly degraded test accuracy, the Latent Optimization attack maintained global accuracy within a narrow margin of the attack-free baseline (e.g., 77.59% vs. 76.75% on CIFAR-10/ResNet-18).
Reshaping Attribution Structure: The attack did not merely boost the attacker's score; it redistributed attribution shares among benign clients, altering the relative credit assignment within the system.
Defense Evasion: Standard geometry-based trimming defenses, which filter updates based on deviation from the bulk of client updates, failed to detect the attack (Precision and Recall near 0.00). The hybrid updates remained aligned with the global descent direction and statistically embedded within the benign distribution.
Robustness: The attack remained effective across different numbers of clients, different initial client rankings, and different attribution evaluators (FedSV and LOO).

Significance and Claims
The paper claims that attribution itself forms a new attack surface in distributed learning. The findings demonstrate that maintaining model accuracy alone is insufficient for trustworthy collaborative learning; attribution integrity is equally critical. The authors argue that current incentive mechanisms and governance frameworks are vulnerable because they rely on evaluators that can be manipulated without harming global performance. This work motivates the development of attribution-robust evaluators and incentive-compatible scoring mechanisms, highlighting that the vulnerability stems from the incentive properties of marginal-utility evaluation rather than specific scoring rules. The paper concludes that without addressing these vulnerabilities, systems risk rewarding the wrong contributors, misidentifying harmful actors, and drawing misleading conclusions about data quality.