💻 computer science

Noise Is Not a Sanitizer: Shared Initialization Turns Data-Free Distillation into a Covert Channel for Memorized Training Data and Backdoors

This paper demonstrates that data-free knowledge distillation using shared initialization fails to sanitize private models, as Gaussian-noise queries can inadvertently transmit memorized training data and backdoors from teacher to student networks.

Original authors: Siddhant Agarwal

Published 2026-10-22

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Siddhant Agarwal

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, models are often trained on vast collections of private information, from medical records to personal messages. To protect this data, researchers have developed a technique called knowledge distillation. Imagine a master chef who has learned to cook from a secret family recipe book. To share the chef's skills without revealing the book, they train a new, smaller apprentice to mimic the master's cooking style using only random, meaningless ingredients like plain water or unseasoned flour. The hope is that because the apprentice never sees the original ingredients, they learn only the general technique and forget the specific secrets. This method has long been considered a safe way to share AI models while keeping the original training data private. It is also used to try to remove hidden "backdoors"—secret triggers planted by attackers that force a model to behave badly—by retraining the model on clean, random data.

However, a new study challenges this long-held belief. Researchers found that if the master and the apprentice start with the exact same basic blueprint, the apprentice can still learn the secrets, even when fed only random noise. The study demonstrates that this process does not sanitize the model as previously thought. Instead, it acts as a hidden pipeline that transfers not just the general skills, but also the specific, memorized details of the private data, including backdoors and private records. This discovery suggests that simply retraining a model on random inputs is not enough to protect privacy if the starting point of the training is known to an outsider.

The researchers tested this idea using image recognition models trained on standard datasets like handwritten digits and clothing items. They created a scenario where a "teacher" model was trained on private data that included special, mislabeled images and hidden triggers designed to act as backdoors. They then attempted to create a "student" model by having it mimic the teacher's responses to random noise, a process known as data-free distillation. Crucially, they compared two scenarios: one where the student started with the exact same initial settings as the teacher, and another where the student started with a completely different, random setup.

The results were striking. When the student and teacher shared the same starting blueprint, the student successfully inherited the teacher's secrets. Even though the student was trained only on random static, it learned to identify the specific private images that the teacher had memorized. In one experiment involving handwritten digits, the student model could correctly guess whether a specific image was part of the teacher's private training set with an AUC of 0.86, a level far above random chance. Furthermore, the student also inherited the backdoor triggers. When presented with images containing a specific hidden pattern, the student model would activate the same malicious behavior as the teacher, with a success rate increase of 93 percent. This happened even when the student was only shown the teacher's internal, unused outputs—signals that were never part of the original training and carried no obvious meaning.

In contrast, when the student started with a different, independent blueprint, it learned nothing of the secrets. The same random noise training produced a model that could not identify the private images or trigger the backdoors. This proves that the leakage was not caused by the random noise itself, but by the shared starting point. The researchers found that this effect was particularly strong when the teacher model had not been trained for too long, staying close to its original state. As the teacher moved further away from its starting point through extensive training, the leak became harder to exploit, but it did not disappear entirely. The study also showed that this vulnerability extends beyond simple image models to language models, where random character strings were enough to extract secret codes that had been inserted into the private training text.

The researchers explained this phenomenon by looking at how the models change during training. They found that when two models share a starting point, the act of mimicking the teacher's responses to random noise effectively pushes the student's internal settings in the same direction as the teacher's. Because the teacher's settings were shaped by the private data, the student's settings are pushed in that same direction, carrying the private information with them. The random noise acts as a vehicle that excites the right parts of the model to reveal these hidden directions. The study showed that this transfer happens regardless of whether the information being moved is useful general knowledge or a specific, memorized secret. The process cannot tell the difference.

To test if this vulnerability could be fixed, the researchers tried several defenses. They found that adding random noise to the answers the teacher gave did not stop the attack; an attacker could simply ask more questions to average out the noise and still recover the secrets. They also found that restricting the output to only meaningless data did not help, as the hidden information was still transferred. The only effective defenses they identified were to either use differential privacy—a mathematical guarantee that limits what any single piece of data can teach the model—or to ensure that the private model starts from a completely different, secret initialization that is not shared with the public. However, the latter option is difficult for models that are fine-tuned from popular public bases, as it requires abandoning the benefits of that public starting point.

This work fundamentally changes how we view the safety of sharing AI models. It shows that the assumption that random inputs act as a filter is incorrect when the models share a common origin. The study concludes that in an era where most AI models are built by fine-tuning a few public base models, the shared starting point is a critical piece of information that attackers can use. Privacy and security evaluations for these models must now consider an adversary who knows the public base and can use random queries to extract private details, proving that data-free distillation is not a safe harbor for sensitive information.

Technical Summary: Noise Is Not a Sanitizer

Problem Statement

The paper challenges the prevailing assumption that data-free knowledge distillation (training a student model on pure noise or synthetic inputs) acts as a privacy sanitizer. It is commonly believed that if a student never sees private training data, it cannot inherit the teacher's memorized records (e.g., mislabeled "canary" examples) or planted backdoors.

The author identifies a critical condition under which this intuition fails: shared initialization. In modern machine learning, private models are frequently fine-tuned from public base checkpoints. If an adversary knows the public initialization ( $\theta_0$ ) used by the private teacher, they can initialize a student model with the same parameters. The author investigates whether this shared starting point creates a covert channel that allows the student to inherit the teacher's instance-level memorization and backdoors, even when trained exclusively on Gaussian noise and potentially only through semantically empty "auxiliary" outputs.

Methodology and Theoretical Framework

Theoretical Analysis: Parameter Transport

The author formalizes the attack using the Neural Tangent Kernel (NTK) and the linearized training regime, where network dynamics are well-approximated by linearization around the initialization.

Theorem 1 (Filtered Parameter Transport): The paper proves that when a student and teacher share an initialization $\theta_0$ , the student's parameter displacement ( $\theta_S - \theta_0$ ) is a linear transformation of the teacher's displacement ( $\theta_T - \theta_0$ ).
$\theta_S(t) - \theta_0 = (I - e^{-\eta t K_Q}) \Delta$
Here, $\Delta$ is the teacher's total parameter change, and $K_Q$ is a channel operator dependent only on the architecture, the query distribution $Q$ , and the initialization. Crucially, the filter $(I - e^{-\eta t K_Q})$ is data-independent. It cannot distinguish between parameter changes encoding generalizable knowledge and those encoding memorized records or backdoors.
Blind Spots (Proposition 1): For one-hidden-layer ReLU networks, the only directions that cannot leak are specific per-neuron rescaling symmetries. Under full-support query distributions (like Gaussian noise), the channel is effectively blind to the semantic content of the displacement, transporting all components at rates determined by their visibility to the queries.
Query Budget and Mismatch: The paper derives that leakage scales with the number of scalar constraints ( $N \times m$ ) relative to parameters ( $p$ ). It also shows that while a small initialization offset ( $\epsilon$ ) masks the signal, a large offset (independent initialization) breaks the linear transport mechanism entirely.

Experimental Setup

The author evaluates this threat across three image datasets (MNIST, Fashion-MNIST, CIFAR-10) and a character-level transformer language model.

Threat Model: An adversary owns the public base checkpoint ( $\theta_0$ ) but has no access to the private training data $D$ . The adversary can query the teacher's API with arbitrary inputs (specifically Gaussian noise) and receives logits.
Attack Strategy: The adversary initializes a student at $\theta_0$ and minimizes the imitation loss on noise queries.
Probes:
1. Membership Inference: Using mislabeled "canary" records inserted into the private training set.
2. Backdoor Extraction: Using a data-poisoning backdoor (a specific trigger patch).
3. Secret Extraction: In the language model, using random six-digit secrets inserted into the fine-tuning text.
Controls: Experiments compare students initialized from the shared $\theta_0$ against those with independent initializations, and against clean teachers (without canaries/backdoors).

Key Results

1. Noise is a Sufficient Query Distribution

Contrary to the belief that noise is uninformative, the experiments show that pure Gaussian noise is sufficient to transmit instance-level memorization.

MNIST/MLP: A student trained solely on noise with a shared initialization achieved a Membership AUC of 0.988 (nearly identical to the teacher's 0.989) and correctly extracted memorized wrong labels for 65.7% of member canaries.
Backdoors: Noise-only distillation transmitted backdoors with an Attack Success Rate (ASR) lift of 99.4%, matching the teacher. Notably, this was more reliable than distillation on clean in-distribution data, which failed to transmit the backdoor in some cases because clean data rarely activates the specific trigger region.

2. The "Subliminal" Auxiliary Channel

A significant finding is that semantically empty outputs leak private information.

When the teacher releases only auxiliary logits (outputs never trained on the private task, carrying no class information), the student still inherits the teacher's memorization.
Results: On MNIST, the auxiliary channel yielded a Membership AUC of 0.86 and a backdoor ASR lift of 93%.
Control: When the student used an independent initialization, the same auxiliary channel carried no signal (AUC $\approx$ 0.53, backdoor lift $\approx$ 14%), proving the leakage relies on the shared initialization, not the outputs themselves.

3. Realistic Fine-Tuning Threat

In a realistic scenario where a provider fine-tunes a public base on private data:

An attacker with the public base and noise queries recovered 92.7% of memorized labels on MNIST and the full backdoor.
On the harder CIFAR-10 dataset, the attack was weaker but still significant: the shared-base attack achieved a calibrated Membership AUC of 0.736 and a backdoor lift of 38.4%, whereas a sibling base (independent initialization) achieved near-chance performance.

4. Language Model Extension

The phenomenon extends to transformers. Random character strings were sufficient to extract secrets (six-digit PINs) fine-tuned into a character-level transformer. The attacker's model achieved an exposure of 19.36 bits (comparable to the private model's 19.65 bits), whereas a sibling base with random queries failed to recover the secrets effectively.

5. Limitations and Boundaries

Teacher Drift: The attack weakens as the teacher moves further from the initialization (e.g., heavily trained CNNs or long training epochs), consistent with the degradation of the linearized-training linear approximation.
Architecture: The auxiliary channel is weaker for Convolutional Neural Networks (CNNs) because isotropic noise induces low activation diversity in convolutional features compared to MLPs. However, full-logit noise attacks remain effective on CNNs.

Defenses Evaluated

The author evaluates several mitigation strategies:

Output Perturbation (Logit Noise): Adding noise to the API responses is ineffective against a data-free attacker. Since the attacker can issue fresh noise queries, they can average out the gradient noise. It only increases the query cost by a constant factor.
Initialization Mismatch: Using a secret offset from the public initialization is effective but costly. The offset must be large ( $\alpha \gtrsim 0.7$ ) to break the channel, which effectively forfeits the utility of the public checkpoint for fine-tuned models.
Differential Privacy (DP): DP-SGD is the only defense that successfully bounds leakage. At $\epsilon \approx 1.5$ , the membership signal and backdoor were effectively removed from both the teacher and the distilled student, though at a cost to model accuracy.

Significance and Claims

The paper claims to fundamentally alter the understanding of data-free distillation as a privacy mechanism.

Rejection of Sanitization: The author argues that "noise is not a sanitizer." The belief that distilling on uninformative inputs removes memorization relies on a function-space view (what the queries reveal). The paper replaces this with a parameter-space view: if the student and teacher share an initialization, the queries only need to excite the parameter directions where private data was written.
Covert Channel: The shared initialization turns the distillation process into a covert channel that transports the teacher's parameter displacement, regardless of whether that displacement encages generalizable knowledge or memorized secrets.
Practical Implication: Public checkpoints are now part of the attack surface. Privacy evaluations of models fine-tuned from public bases must assume an adversary who owns the base checkpoint. "Data-free" does not equate to "privacy-preserving" in the presence of shared initialization.

The paper concludes that effective defenses must act on the parameters (e.g., Differential Privacy or large initialization offsets) rather than on the output interface (e.g., logit noise or restricting output channels).