Towards Distillation-Resistant Large Language Models: An Information-Theoretic Perspective

1. Problem Statement

Large Language Models (LLMs) represent significant intellectual property (IP) and economic value. While providers typically expose these models via black-box APIs to protect their weights, adversaries can still extract model capabilities through Knowledge Distillation (KD).

• The Gap: Existing defenses primarily focus on text-based (sequence-level) distillation, where attackers use the sampled output tokens as pseudo-labels. However, logit-based (response-level) distillation remains largely unaddressed.
• The Challenge: Logit-based KD exploits the full probability distribution (logits) of the teacher model, which contains rich contextual information beyond the final predicted token. This allows student models to learn finer-grained patterns, making the theft of capabilities more efficient and the defense significantly harder. Current methods that perturb token distributions often degrade the quality of the teacher's output, rendering them impractical.

2. Methodology

The authors propose an information-theoretic approach to defend against logit-based KD by minimizing the "distillation-relevant" information in teacher outputs while preserving task utility.

A. Theoretical Foundation: Conditional Mutual Information (CMI)

The core insight is that the information useful for distillation corresponds to the contextual information in the teacher's logits ($Z$) that is not explained by the ground-truth label ($Y$).

• Metric: The authors define the defense objective as minimizing the Conditional Mutual Information (CMI): $I(X; Z | Y)$.
  • $X$: Input query.
  • $Y$: Ground-truth label (next token).
  • $Z$: Teacher logits.
• Interpretation: $I(X; Z | Y)$ measures how much information the logits $Z$ reveal about the input $X$ given the label $Y$. Minimizing this quantity suppresses the contextual cues that attackers exploit, effectively "purifying" the logits to contain only the necessary label information.

B. Defense Mechanism: Learnable Transformation Matrix

To achieve CMI minimization without retraining the massive teacher model, the authors introduce a lightweight, post-processing learnable transformation matrix $M$.

• Process: The original logits $Z$ are transformed into refined logits $Z' = M \cdot Z$.
• Theoretical Guarantee: The paper proves (Theorem 2) that under a Markov chain $Y \to X \to Z \to Z'$, the transformation satisfies $I(X; Z | Y) \geq I(X; Z' | Y)$, guaranteeing that the transformation reduces distillation-relevant information.

C. Optimization Objective

The authors derive a practical loss function based on the Information Bottleneck (IB) principle to optimize $M$. The objective decomposes into two complementary terms:

1. Preserve Utility (Maximize $I(Z'; Y)$):

   • Goal: Ensure the transformed logits still predict the correct label.
   • Implementation: Minimize the Cross-Entropy Loss ($L_{CE}$) between the transformed logits $Z'$ and the ground-truth label $Y$. This serves as a tractable upper bound for maximizing mutual information.

2. Suppress Context (Minimize $I(X; Z')$):

   • Goal: Remove information about the input $X$ from the logits.
   • Challenge: Directly estimating $I(X; Z')$ is intractable for high-dimensional LLMs.
   • Solution: The authors observe that in KD, the information from $X$ is transmitted to the student via gradients. They propose maximizing the angular deviation between the gradients induced by the original logits ($g$) and the transformed logits ($g'$).
   • Implementation: A gradient deviation loss ($L_{grad}$) that maximizes the angle (minimizes cosine similarity) between $g$ and $g'$. This "poisons" the learning signal for the student without altering the teacher's final output distribution significantly.

Total Loss: $L_M = L_{CE} + \lambda \cdot L_{grad}$

D. Efficient Implementation

• Parameter Efficiency: Since the vocabulary size $|V|$ is large, a dense $|V| \times |V|$ matrix is computationally prohibitive. The authors decompose $M$ into a low-rank structure: $M = E + AB$, where $E$ is the identity matrix, and $A, B$ are low-rank matrices (initialized via LoRA-style initialization).
• Surrogate Student: A small proxy student model is used during the training of $M$ to compute the gradients $g$ and $g'$ for the $L_{grad}$ term.

3. Key Contributions

1. First Defense Against Logit-Based KD: The paper is the first to specifically address and defend against logit-based knowledge distillation, a more potent attack vector than text-based KD.
2. Information-Theoretic Framework: It introduces CMI as a principled metric to quantify and minimize the leakage of distillation-relevant information.
3. Novel Optimization Strategy: It proposes a dual-loss objective (Cross-Entropy + Gradient Deviation) that effectively purifies logits while maintaining high task accuracy.
4. Lightweight Post-Processing: The method requires only a small, learnable matrix added to the output layer, leaving the massive teacher model frozen and preserving inference efficiency.

4. Experimental Results

The method was evaluated across multiple LLM families (Qwen2.5, Llama 3) and datasets (GSM8K, MMLU, MATH) against four strong distillation baselines (Vanilla KD, AlphaNet, MiniLLM, ABKD).

• Defense Efficacy: The proposed method significantly degrades the performance of distilled student models.
  • Example: On GSM8K, distilling a Qwen2.5-1.5B student using AlphaNet dropped accuracy from 62.93% (Vanilla) to 50.95% (Defense), a drop of ~12%.
  • In many cases, the distilled student performed worse than a model trained only on ground-truth labels (SFT-only), indicating the defense successfully disrupted the distillation signal.
• Utility Preservation: The teacher model's accuracy remained largely intact (e.g., dropping only marginally from 62.93% to 62.85% in some configurations), ensuring the API remains useful for legitimate users.
• Robustness: The defense worked consistently across different student model sizes (0.5B to 3B) and different distillation algorithms.
• Qualitative Analysis:
  • Teacher: Outputs remained fluent, coherent, and correct.
  • Student: Distilled students exhibited degraded reasoning, shorter responses, incomplete derivations, and in some cases, repetitive loops or generation collapse, confirming the disruption of the knowledge transfer mechanism.

5. Significance

This work represents a pivotal shift in LLM security. By moving beyond post-hoc watermarking and text-level perturbations, it addresses the root cause of logit-based theft: the excess information in probability distributions.

• IP Protection: It provides a robust mechanism for commercial providers to protect their models from "model stealing" via API access, a critical concern as LLMs become more valuable.
• Theoretical Insight: It bridges information theory and LLM security, offering a new perspective on how to quantify and control information leakage in generative models.
• Practicality: The low-rank, post-processing nature of the defense makes it highly deployable in real-world API scenarios without the need for expensive model retraining.