🤖 machine learning

Lossless Anti-Distillation Sampling

This paper proposes Lossless Anti-Distillation Sampling (LADS), a novel scheme that generates responses using query-dependent private seeds to preserve perfect quality for benign users while intentionally introducing correlated randomness across multiple accounts to degrade the performance of models trained via distillation.

Original authors: Zibo Diao, Jingchu Gai, Xinyue Ai, Zhang Zhang, Zhenyu He, Di He

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zibo Diao, Jingchu Gai, Xinyue Ai, Zhang Zhang, Zhenyu He, Di He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Copycat" Thief

Imagine a famous chef (the Teacher Model) who runs a high-end restaurant. This chef creates amazing, unique dishes based on secret recipes.

A competitor (the Distiller) wants to steal the chef's success without doing the hard work of inventing new recipes. Instead of hiring a team of chefs to learn from scratch, the competitor hires 50 different people (the Multi-Accounts) to order the exact same dishes from the famous chef's restaurant. They write down every ingredient and step, then use that list to train their own "student" chef to cook the same food.

Because the competitor uses 50 different people, the famous chef's security system doesn't flag them as suspicious; each person just looks like a normal customer. The thief gets a massive amount of data for free, and the original chef loses their competitive edge.

The Old Solutions (And Why They Failed)

Previously, chefs tried two ways to stop this:

The "Bad Food" Tactic: The chef intentionally adds a little bit of salt or changes the recipe slightly for everyone. This makes it hard for the thief to copy the exact taste, but it also ruins the meal for the honest customers.
The "Security Guard" Tactic: The chef watches the customers closely. If someone orders 100 times, they get kicked out. But the thief just uses 50 different people to order 2 times each, fooling the guard.

The New Solution: LADS (The "Secret Coin Flip")

The authors propose a new method called Lossless Anti-Distillation Sampling (LADS).

Instead of changing the food (the output), they change the randomness behind the cooking process.

The Analogy: The Magic Coin Flip
Imagine that every time a customer orders a dish, the chef flips a magic coin to decide a tiny, invisible detail of the meal (like the exact temperature of the oven or the precise timing of a garnish).

For a normal customer: Every time they visit, the chef flips a new, independent coin. The customer gets a unique, high-quality meal every time. They never notice anything is different.
For the thief with 50 accounts: The chef has a secret rule. If the thief's 50 different people order the same type of dish (e.g., "Spicy Ramen") and they are all on their first visit, the chef secretly uses the same coin flip for all 50 of them.

The Result:

The Honest Customer: Gets a perfect, unique meal every time. They are happy.
The Thief: They think they have collected 50 different recipes. But because the chef used the same "coin flip" for all 50 accounts, the thief actually only has one unique recipe repeated 50 times.

Why This Stops the Thief

In machine learning, to learn well, a student needs to see many different examples (diversity).

If the thief collects 50 different variations of "Spicy Ramen," their student chef learns to make great ramen.
If the thief collects 50 identical variations (because the chef forced the same randomness), the student chef learns nothing new. They just memorize one specific version of the dish.

The paper proves mathematically that by forcing this "shared randomness" across different accounts, the thief's student model becomes much worse at generalizing. It's like trying to learn to swim by watching the same person swim the exact same stroke 50 times, rather than watching 50 different people swim.

The "Lossless" Part

The most important part of this paper is that honest users don't lose anything.

Because the thief's accounts are just pretending to be normal, the "shared coin flip" only happens when the thief tries to game the system.
A real person who visits the restaurant once a week gets a fresh, random coin flip every time. Their experience is 100% perfect and unchanged.

Summary of the Experiments

The authors tested this idea in two real-world scenarios:

Image Generation: They used an AI that draws pictures. When the "thief" tried to steal the drawing style using 50 fake accounts, the thief's student AI produced blurry, low-quality images. Meanwhile, real users still got beautiful, sharp pictures.
Language Models (Math & Code): They used an AI that solves math problems and writes code. When the thief tried to steal the AI's brain using multiple accounts, the thief's student AI got much worse at solving math problems and writing code. Real users, however, still got perfect answers.

The Bottom Line

LADS is a clever trick that protects AI models from being stolen by "copycat" thieves using multiple fake accounts. It does this by secretly linking the randomness of the AI's answers together for similar requests. This makes the stolen data useless for training a new model, while ensuring that regular, honest users continue to get the best possible experience.

Technical Summary: Lossless Anti-Distillation Sampling (LADS)

Problem Definition

Frontier commercial generative models face a significant threat from distillation, where adversaries harvest generated responses via API queries to train competing models at a fraction of the original training cost. While defenses exist for single-account attacks (e.g., monitoring usage volume), they are ineffective against multi-account distillation, where adversaries distribute queries across hundreds of accounts, each mimicking benign user behavior.

Existing countermeasures suffer from critical limitations:

Output Perturbation: Methods that alter model outputs to poison gradients (e.g., Prediction Poisoning, Nasty Teacher) inherently degrade response quality for benign users.
Watermarking: Post-hoc detection methods (e.g., GINSEW, Antidistillation Fingerprinting) do not prevent the distillation process itself and often alter the output distribution.
Behavioral Detection: Adaptive defenses that trigger only upon detecting anomalies can be circumvented by distributing queries across many accounts that individually appear benign.

The core challenge is to design a defense that is lossless for benign users (preserving exact statistical fidelity) while effectively degrading the performance of a multi-account distiller.

Methodology: Lossless Anti-Distillation Sampling (LADS)

The authors propose LADS, a server-side mechanism that controls the realization of the underlying noise variable $\epsilon$ used in generation, rather than modifying the model's output distribution $G$ .

Core Mechanism

LADS derives the random seed for each generation from a private injective seed generator $SG$, which takes two inputs:

Semantic Bucket ( $i$ ): A private clustering function $h(q)$ maps the query $q$ to a semantic bucket. In practice, this can be realized via Locality-Sensitive Hashing (LSH) on semantic embeddings.
Bucket-Local Visit Counter ( $a[i]$ ): The number of times the specific user has queried within that semantic bucket.

The seed is computed as $s = SG(i, a[i])$. This seed initializes a Pseudorandom Number Generator (PRNG) to produce the noise $\epsilon$ , which is then used to generate the output $x = G(q, \epsilon)$ .

Operational Properties

Losslessness for Benign Users: For any single account, the sequence of seeds across different requests is unique (due to the incrementing counter). Consequently, the user receives samples that are statistically independent and identically distributed (i.i.d.) according to the original model's distribution. No distortion is introduced.
Correlation for Distillers: A multi-account distiller controlling $K$ accounts will submit semantically similar queries (falling into the same bucket $i$ ) across different accounts. If these queries occur at the same relative access count (e.g., the $t$ -th query in the bucket for all accounts), they are assigned the same seed. This forces the generation of correlated outputs across accounts.
Impact on Distillation: Instead of receiving $KT$ independent samples, the distiller effectively receives only $T$ independent samples (each repeated $K$ times). This reduces the effective sample size and introduces statistical correlation that hinders the student model's ability to generalize.

Theoretical Framework

The paper formalizes the threat as a distiller minimizing empirical risk over $K$ accounts with $T$ queries each.

Unconditional Generation (LADS-SIMPLE): Without query conditioning, the generalization gap under standard i.i.d. sampling scales as $O(1/\sqrt{KT})$ . Under LADS-SIMPLE, it degrades to $O(1/\sqrt{T})$ , effectively nullifying the benefit of adding more accounts.
Conditional Generation (LADS): For task-specific distillation, the generalization gap depends on the number of activated semantic buckets $N$ rather than the number of accounts $K$ . The bound scales as $O(1/\sqrt{NT})$ . Since $N \ll K$ (semantic buckets are fewer than accounts), the convergence rate is significantly slower than standard i.i.d. sampling. The analysis utilizes uniform convergence theory and Rademacher complexity to prove that the cross-account noise coupling reduces the statistical efficiency of the distillation process.

Experimental Results

The authors evaluated LADS on two paradigms: diffusion-based image generation and autoregressive language models.

1. Image Generation (EDM2 Models)

Setup: Teachers (EDM2-Large, EDM2-XS) vs. Student (EDM2-XXS). Multi-account simulation with 50 accounts querying ImageNet classes.
Metric: Fréchet Inception Distance (FID) and validation loss.
Findings: As the noise coupling strength $\alpha$ $α$ increased from 0 (standard i.i.d.) to 1 (full LADS), the student's performance degraded substantially.
- FID (to real) for EDM2-Large increased from 8.50 ( $\alpha=0$ ) to 29.8 ( $\alpha=1$ ).
- Validation loss increased monotonically with $\alpha$ .
- Benign users (simulated by a single account) maintained high-quality FID scores comparable to the original model, confirming the lossless property.

2. Language Model (Math and Code)

Setup: Teacher (Qwen3.5-397B) vs. Students (Llama-3.1-8B, DeepSeek-Math-7B, Qwen2.5-7B). Benchmarks included MATH, GSM8K, and HumanEval.
Method: The teacher generated responses for prompts and their paraphrases. In the LADS setting, semantically equivalent prompts shared the same random seed.
Findings:
- Standard i.i.d. Sampling: Fine-tuning yielded significant gains (e.g., +29.8 points on GSM8K for Llama-3.1-8B).
- LADS Sampling: Gains were drastically reduced (e.g., +11.2 points on GSM8K for the same model).
- The results confirm that LADS limits the statistical benefit of multi-account distillation while preserving the quality of individual responses.

Key Contributions

Novel Defense Mechanism: Introduction of LADS, the first defense that achieves lossless protection against multi-account distillation by manipulating the underlying noise realization rather than the output distribution.
Theoretical Guarantee: A rigorous proof using uniform convergence theory demonstrating that LADS collapses the effective sample size from $KT$ to $T$ (or $NT$ in conditional settings), provably degrading the distiller's generalization gap.
Empirical Validation: Comprehensive experiments across image and language domains showing that LADS significantly degrades student model performance while maintaining exact statistical fidelity for benign users.

Significance and Claims

The paper claims that LADS addresses a critical gap in AI security: the inability to defend against sophisticated, multi-account distillation attacks without compromising the user experience. By shifting the defense from the output space to the randomness space, the authors argue that it is possible to make distillation "statistically inefficient" for adversaries while remaining invisible to legitimate users.

The authors position LADS as a principled alternative to output perturbation and watermarking, offering a proactive defense that prevents the acquisition of high-quality training data rather than merely detecting unauthorized use after the fact. They note that future work could involve more adaptive semantic grouping strategies and integration with system-level protections, but the current work focuses on establishing the theoretical and empirical viability of noise-coupling as a defense mechanism.