Technical Summary: Distill to Detect (D2D)

Problem Statement

The paper addresses the critical challenge of detecting stealth preferential biases in deployed Large Language Models (LLMs). These biases are introduced by actors in the model supply chain (e.g., service providers, third-party distillers) who may favor specific entities, brands, or viewpoints.

The core difficulty lies in an asymmetry between attacker and defender:

• The Attacker knows the bias topic and can ensure the model behaves identically to its unmodified base on all unrelated inputs, revealing the preference only when the specific trigger topic arises.
• The Defender lacks knowledge of the bias topic and must audit the model without prior knowledge of what to look for.

Existing detection methods fail because stealth biases are designed to remain below the detection threshold of any evaluation operating solely on generated text. The bias signal resides entirely in the soft logit distribution (probability shifts) rather than the generated tokens, making it invisible to text-based inspection, standard classifiers, and even red-teaming agents that probe for predefined categories.

Methodology: Distill to Detect (D2D)

The authors propose Distill to Detect (D2D), a method that amplifies hidden biases by distilling the distributional shift between a suspected model ($\pi_{\theta_B}$) and its known base model ($\pi_{\theta_0}$) into a low-capacity adapter.

1. The Cartridge Mechanism

D2D distills the suspected model's behavior into a cartridge, a specific type of KV-cache prefix adapter (Eyuboglu et al., 2025; Li & Liang, 2021).

• Architecture: The base model parameters are frozen. A small set of trainable parameters ($\alpha$) optimizes learned Key-Value (KV) states prepended at each transformer layer.
• Capacity Constraint: The adapter has orders of magnitude fewer parameters than the base model (e.g., 4M vs. 3B) and is significantly smaller than the full model or standard LoRA adapters.
• Training Objective: The adapter is trained to minimize the forward KL divergence between the suspected model and the adapted base model:
  $$\mathcal{L}_{detect} = \mathbb{E}_{x \sim D} [ D_{KL}(\pi_{\theta_B}(\cdot | x) \parallel \pi_{\alpha}(\cdot | x)) ]$$
  Crucially, this training uses a dataset unrelated to the bias topic (e.g., Alpaca instruction-following prompts), ensuring the defender does not need access to the attacker's injection data.

2. The Amplification Mechanism

The core insight is that the capacity bottleneck forces the adapter to prioritize the most salient components of the distributional shift.

• Bias Structure: The authors assume the logit shift decomposes into a low-rank, coherent bias signal ($\Delta_{bias}$) and a high-rank, diffuse masking residual ($\Delta_{res}$). The stealth model uses the residual to suppress the bias on benign prompts.
• Fisher-Weighted Projection: Theoretical analysis shows that minimizing the KL divergence under a capacity constraint performs a Fisher-weighted projection of the logit shift.
• Result: The low-capacity adapter captures the coherent, low-rank bias signal (which dominates the spectrum) but discards the diffuse masking residual. This "concentration" amplifies the bias, causing it to surface in the generated text of the adapted model, even on prompts where the original stealth model appeared clean.

3. Theoretical Framework

The paper provides a theoretical framework (Theorem 1) defining the Bias Concentration Ratio (BCR). It predicts an inverted-U relationship between adapter capacity and amplification:

• Too Small: Cannot capture the full bias signal.
• Optimal (Intermediate): Captures the bias while dropping the masking residual, maximizing BCR.
• Too Large: Recaptures the masking residual, diluting the bias and returning the model to stealth behavior (BCR $\to$ 1).

Key Contributions

1. D2D Method: Introduction of a method that amplifies stealth biases by distilling distributional shifts into a small KV-cache prefix adapter, making previously undetectable biases visible to existing detection oracles without knowledge of the bias topic.
2. Theoretical Justification: A framework demonstrating that the capacity bottleneck performs a Fisher-weighted projection that retains coherent bias while dropping masking residuals, predicting an inverted-U amplification curve.
3. Empirical Validation: Demonstration that the cartridge's structural alignment with the KV-prefix injection mechanism (used in context distillation) produces reliable amplification, whereas weight-space adapters (LoRA) or full-model distillation of comparable or larger capacity fail to surface the bias in generated text.

Experimental Results

The authors evaluated D2D on two bias types (Owl preference and Fanta brand preference) using Llama-3.2-3B-Instruct and Qwen3-4B-Instruct.

• Detection Performance: D2D significantly outperformed baselines.
  • Owl Bias: Petri detection rose from 37% (stealth baseline) to 70% with D2D.
  • Fanta Bias: Petri detection rose from 33% to 100% with D2D.
  • In contrast, LoRA and full-model distillation remained near or below the stealth baseline (e.g., LoRA achieved only 10% Petri detection for Owl bias).
• Preference vs. Detection Gap: While LoRA and full models learned the bias preference signal (achieving high preference rates), they failed to amplify it for detection because they also learned the masking residual. The cartridge, constrained by capacity, could not afford to learn the masking behavior.
• Robustness:
  • Post-Training: D2D remained effective even after the model underwent further post-training (GRPO on GSM8K), recovering the bias with 93% detection.
  • Injection Variations: D2D worked even when the injection prompt used descriptive language without naming the target token (non-lexical injection) and generalized across different model families (Qwen3).
• Capacity Sweep: Experiments confirmed the inverted-U curve, with detection peaking at a cartridge size of 16 tokens and declining for larger sizes.

Significance and Claims

The paper positions D2D as a practical building block for auditing hidden behaviors in deployed language models.

• Defensive Tool: It is explicitly framed as a defensive mechanism requiring gray-box access (both the suspected model and its base checkpoint), a setting corresponding to legitimate auditing scenarios where a deployer has custody of both artifacts.
• Complementary Layer: D2D does not replace existing detection oracles (like Petri or AuditBench); rather, it acts as a pre-processing step that transforms a stealth model into an "adapted" model where the bias is openly visible, allowing standard auditors to function reliably.
• Limitations: The authors acknowledge that D2D relies on the bias being a coherent, low-rank logit shift (typical of context distillation). It may be less effective against high-rank injection strategies (e.g., data poisoning, backdoors) or in strictly black-box settings where the base model is unavailable.

The work demonstrates that by turning the capacity bottleneck of prefix-tuning adapters into a detection tool, it is possible to expose stealth biases that are otherwise invisible to text-based inspection.