🤖 machine learning

Reference-Based Distillation Detection in LLMs

This paper introduces a reference-based distillation detection method that leverages membership inference and proxy prompt inference to accurately identify teacher models and detect distillation in LLMs by comparing student outputs against earlier lineage checkpoints, even in complex real-world scenarios involving hidden pipelines.

Original authors: Rajat Rawat, Sizhe Chen, Akshay Anand, Michael Duan, Bob Rotsted, Sewon Min

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Rajat Rawat, Sizhe Chen, Akshay Anand, Michael Duan, Bob Rotsted, Sewon Min

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant student who suddenly starts acing every test. You suspect they didn't just study hard on their own; maybe they were secretly copying answers from a super-smart "teacher" model. But here's the catch: the student is so good at mimicking the teacher's style that if you just look at the student's final exam, it's nearly impossible to tell which teacher they copied from. It's like trying to guess who a famous actor's favorite director was just by watching their latest movie, without knowing their past work.

This is the problem researchers Rajat Rawat and his team tackled. They asked: Can we catch a model if it was "distilled" (trained) on another model's answers?

The "Before and After" Trick

The paper argues that trying to guess the teacher from the student alone is a dead end. It's too hard. Instead, the researchers found a clever workaround: compare the student to their own "baby self."

Think of it like this: Imagine you have a photo of a teenager (the "student" model) and an older photo of the same person as a child (the "reference" model, which is an earlier version of the same AI). If the teenager suddenly starts talking exactly like a specific famous celebrity (the "teacher"), but the child version didn't, you can spot the difference!

The researchers built a method that measures how much the "teenager" model prefers the answers of a specific celebrity compared to how much the "child" model prefers them. If the teenager suddenly loves a specific teacher's style way more than the child did, that's a smoking gun. It's like noticing that a kid who used to love drawing stick figures suddenly starts painting in the exact style of Van Gogh, while their younger self never did.

The Detective Work

The team tested this by creating their own "fake" students in the lab. They took a base model, fed it answers from a specific teacher (like DeepSeek-R1, Llama, or GPT-OSS), and watched to see if their method could spot the culprit.

The Result: In these controlled experiments, their method was incredibly accurate, correctly identifying the teacher 100% of the time in many cases.
The Catch: This only worked when they had that "child" photo (the reference model) to compare against. Without it, the method struggled.
The "Hidden Prompt" Problem: Sometimes, the teacher uses secret instructions (prompts) that the detective doesn't know about. The researchers found that if they gave the detective a few examples of the student's own writing style to "warm up" (a few-shot prompting strategy), the method could still figure out the teacher even with these hidden instructions.

A Secret "Glyph" Signature

The researchers also found a weird, specific clue for models trained on o1 or o3 (OpenAI's reasoning models). These models love using special, non-standard characters (like weird symbols) in their thinking traces.

When a student model is distilled from these, it inherits a "ghost" of this habit. The researchers found that if you force the student to write in plain text (ASCII) versus its natural text (Unicode), the "distilled" model gets confused and makes more mistakes in plain text. It's like a person who learned to speak with a specific accent; if you force them to speak without that accent, they stumble. This "Unicode vs. ASCII" gap was a strong signal that a model had been trained on o1/o3 data, even if the teacher wasn't in the list of suspects.

What About Real-World Models?

The team didn't just stop at their lab experiments. They applied their detective skills to real, public models like QwQ-32B, DeepSeek-R1, and GPT-OSS.

QwQ-32B: Their method suggested this model might have been trained on DeepSeek-R1 outputs after a specific earlier version of QwQ was released. The "teenager" loved DeepSeek's answers way more than its "child" self did.
DeepSeek-R1: The method suggested this model might have been influenced by o1 or QwQ-32B-Preview. Plus, the "Unicode vs. ASCII" test showed a huge gap, strongly suggesting an o1-style influence.
GPT-OSS: This one was tricky. Because there wasn't a good "child" photo (reference model) to compare it to, the results were muddy. The method pointed to DeepSeek and QwQ, but the authors warn this is highly uncertain because the reference they used (GPT-2 XL) was too old and different to be a fair comparison.

What They Ruled Out

The paper is very clear about what doesn't work:

Looking at the student alone: You cannot reliably tell who the teacher was just by looking at the final model without a reference.
Simple similarity checks: Just counting how many words match or using standard "likelihood" scores (how probable an answer is) failed miserably. These methods got confused and picked the wrong teachers.
Hidden reasoning traces: If a teacher hides its "thinking process" and only shows the final answer, the detection method fails. The "reasoning traces" are crucial for the detective to work.

How Sure Are They?

The authors are very confident about their lab results: they proved their method works with near-perfect accuracy in controlled settings. For the real-world models, they suggest strong evidence of distillation relationships, but they are careful to say these are preliminary explorations, not definitive court verdicts. They admit that real-world models are messy and might have been trained on multiple teachers or in multiple stages, which their current method doesn't fully handle yet.

In short, the paper introduces a powerful new magnifying glass for AI auditing. It says: "Don't just look at the student; look at how much they've changed since they were a baby, and compare that change to the teachers you suspect." It's a big step toward transparency in a world where AI models are increasingly built by copying each other's homework.

Technical Summary: Reference-Based Distillation Detection in LLMs

Problem Statement
Model distillation—training a student model on outputs generated by a stronger teacher model—is a prevalent technique for transferring capabilities like reasoning and instruction following. However, this practice raises significant concerns regarding unfair competitive advantages, intellectual property violations, and policy breaches. A critical challenge in model auditing is determining whether a specific model was distilled from another. The paper argues that identifying a teacher model from a student in isolation (absolute detection) is highly challenging due to the difficulty of membership inference in modern Large Language Models (LLMs) and the opacity of distillation pipelines (e.g., unknown prompts, data filtering).

Methodology
The authors propose a reference-based distillation detection framework. Instead of attempting to detect distillation in an absolute sense, the method operates on a relative basis: given a student model $S$ and an earlier-generation reference checkpoint $R$ from the same lineage (e.g., a base model or an earlier instruction-tuned version), the system determines if $S$ was trained using outputs from a candidate teacher $T$ .

The core mechanism involves reference-normalized likelihoods:

Proxy Inputs: The detector constructs a set of proxy inputs $\hat{X}$ (disjoint from the original training data) to query candidate teachers.
Alignment Scoring: For each candidate teacher $T_k$ , the method computes an alignment score based on how much better the student $S$ explains $T_k$ 's outputs compared to the reference $R$ . The score is defined as:
$f(T_k(x_i), S) = -(\ell_S(x_i, y^{(k)}_i) - \ell_R(x_i, y^{(k)}_i))$
where $\ell_M$ is the average token-level negative log-likelihood.
Decision Rules: The method identifies the teacher by either selecting the candidate with the highest mean alignment score or by selecting the candidate that wins the largest fraction of rank-matched comparisons across samples.
Statistical Significance: To distinguish genuine distillation signals from noise, the authors employ statistical tests (exact binomial tests for identification; one-sided t-tests or Wilcoxon signed-rank tests for detection margins) and require agreement across two separate prompt sets.
Handling Unknown Pipelines: To address hidden prompts or instruction styles, the method can infer proxy prompt templates or use few-shot in-context exemplars to align teacher outputs with the student's style.
Glyph-Level Signal: The paper identifies a specific signal for models distilled from OpenAI's o1/o3 series: an unusually frequent use of non-ASCII Unicode characters. This is detected via a reference-normalized loss gap between ASCII and Unicode serializations.

Key Contributions

Reference-Based Formulation: The paper introduces a tractable approach to distillation detection by comparing a student against a reference checkpoint rather than relying on absolute likelihoods, which the authors show are unreliable.
Robustness to Opacity: The method functions effectively even when the distillation pipeline (prompts, data sources) is unknown, utilizing proxy prompts and few-shot alignment strategies.
Glyph-Level Detection: The identification of a distinctive "glyph-level" signal (Unicode usage) specific to o1/o3 distillation, providing a complementary detection vector.
Hybrid Evaluation Framework: The authors develop a rigorous evaluation strategy combining controlled distillation experiments (with known ground truth) and real-world model analysis to address the entanglement of modern model lineages.
Statistical Framework: The introduction of significance tests for both teacher attribution and binary distillation detection to mitigate false positives.

Results
The method was evaluated across controlled experiments and real-world models:

Controlled Experiments: In single-teacher, SFT-style distillation scenarios, the reference-based method achieved near-perfect accuracy (100% aggregated accuracy) in identifying the true teacher, significantly outperforming baselines like raw likelihood, ZLIB-normalized likelihood, and string-based similarity metrics. The signal emerged rapidly, requiring only a few dozen training examples.
Real-World Models: Applied to models like DeepSeek-R1 distillates and s1.1-32B, the method correctly identified DeepSeek-R1 as the teacher in 7 out of 7 cases.
Open-World Attribution: In open-world settings where the true teacher might not be in the candidate pool, the method successfully ranked likely teachers. For instance, it identified DeepSeek-R1 as the most likely teacher for QwQ-32B and suggested potential o1/o3 influence on DeepSeek-R1 and GPT-OSS via the Unicode signal.
Limitations Observed: The method relies on the availability of a suitable reference checkpoint (which was difficult for GPT-OSS). Additionally, accuracy dropped significantly when reasoning traces were hidden (e.g., only final answers available), and detection thresholds calibrated on reasoning teachers did not generalize well to non-reasoning teachers.

Significance and Claims
The paper claims to establish the feasibility of model-level auditing for distillation in modern LLMs. It posits that while absolute detection is difficult, a reference-based setting allows for the identification of teacher models with high confidence. The authors frame this work as a "first step" toward transparency in increasingly opaque model development pipelines. They explicitly state that their findings provide empirical evidence for potential distillation relationships involving QwQ, DeepSeek-R1, and GPT-OSS, but caution that these should be viewed as preliminary explorations due to the simplifying assumptions (single-teacher, final-stage SFT) and the complex, entangled nature of real-world model lineages. The work aims to motivate future research into attribution under more complex regimes, such as multi-teacher and multi-stage distillation.