💬 NLP

Intrinsic Fingerprint of LLMs: Continue Training is NOT All You Need to Steal A Model!

This paper proposes a robust LLM fingerprinting method based on the stable standard deviation distributions of attention parameter matrices, demonstrating that these intrinsic signatures can reliably identify model lineage and detect copyright infringement even after extensive continued training or upcycling.

Original authors: Do-hyeon Yoon, Minsoo Chun, Thomas Allen, Hans Müller, Min Wang, Rajesh Sharma

Published 2026-04-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Do-hyeon Yoon, Minsoo Chun, Thomas Allen, Hans Müller, Min Wang, Rajesh Sharma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The "DNA Test" for AI: How to Catch a Model Thief

Imagine you are a world-class chef. You spend years perfecting a secret, complex sauce. It’s your masterpiece, and it costs you a fortune in rare spices and labor to create. One day, a rival restaurant opens up across the street. They claim they invented their own "signature sauce" from scratch, but when you taste it, it’s exactly like yours—just a little bit spicier.

How do you prove they didn't invent it? You can't just look at the taste, because they might have added extra pepper to hide the truth. You need something deeper. You need to look at the microscopic structure of the molecules in the sauce.

This paper is essentially a "DNA test" for Artificial Intelligence.

The Problem: The "Identity Theft" of AI

Right now, building massive AI models (like ChatGPT or Qwen) is incredibly expensive—it costs millions of dollars in electricity and supercomputers. Because it's so expensive, some companies are tempted to "steal" a model that someone else already built, tweak it slightly, and then claim, "Look! We built this ourselves from scratch!"

In the AI world, this is called "Upcycling" or "Continued Training." It’s like taking someone else’s finished cake, adding some frosting, and telling everyone you baked the whole thing from flour and eggs.

The Old Way: The "Watermark" (The Fake Signature)

Previously, people tried to protect AI by "watermarking" it—essentially asking the AI to leave a specific "signature" in the text it writes.

But this is easy to beat. It’s like a person signing their name on a painting. If a thief wants to steal the painting, they can just paint over the signature. If you "continue training" an AI (the equivalent of painting over the signature), the watermark disappears, and the thief gets away with it.

The New Discovery: The "Intrinsic Fingerprint"

The researchers in this paper found something much smarter. They realized that when an AI is being trained, the mathematical "weights" (the internal settings) inside its brain develop a very specific pattern.

They focused on the Attention Mechanism—the part of the AI that decides which words are important. They discovered that if you look at the "standard deviation" (a math term for how much the numbers vary) of these settings across different layers of the AI, it creates a unique, wavy pattern.

Think of it like a fingerprint or a mountain range. Even if you change the color of the mountains (fine-tuning the model) or turn a single mountain into a range of smaller hills (changing the architecture), the fundamental "shape" of the landscape remains the same. This shape is "intrinsic"—it’s baked into the very bones of the model.

The "Smoking Gun": The Huawei vs. Qwen Case

To prove their method works, the researchers did a real-world investigation.

They looked at a model called Pangu Pro MoE (released by Huawei) and compared it to a model called Qwen-2.5 14B (released by Alibaba/Qwen). Huawei claimed they trained Pangu on a massive amount of data (13 trillion tokens).

However, when the researchers ran their "DNA test," the results were shocking:

The "fingerprints" of Pangu and Qwen were almost identical.
The mathematical correlation was incredibly high (over 92%).

It’s like finding out that a "new" brand of bottled water actually has the exact same mineral fingerprint as a famous brand, despite the company claiming they dug their own well. The researchers are suggesting that Pangu wasn't built from scratch; it was likely "upcycled" from Qwen.

Why This Matters

This paper is a warning shot to the AI industry. It tells companies: "You can't hide your tracks by just training a little bit more."

As AI becomes the backbone of our economy, we need ways to ensure that creators get credit for their work and that companies are being honest about how they build their technology. This "fingerprinting" method provides a way to hold the giants accountable, ensuring that innovation is rewarded and plagiarism is caught.

Technical Summary: Intrinsic Fingerprint of LLMs

Title: Intrinsic Fingerprint of LLMs: Continue Training is NOT All You Need to Steal A Model!
Authors: Do-hyeon Yoon, Minsoo Chun, Thomas Allen, Hans Müller, Min Wang, and Rajesh Sharma (Honest AGI Community)

1. Problem Statement

As Large Language Models (LLMs) become increasingly expensive to train, the risk of intellectual property (IP) theft, unauthorized model reuse, and "model plagiarism" has grown. Traditional protection methods, such as watermarking, are often "brittle"—they embed signals in the model's output that can be easily erased or obscured through continued training (fine-tuning), distillation, or paraphrasing attacks. There is a critical need for a robust fingerprinting method that is tied to the model's internal architecture and remains detectable even after significant modifications or additional training.

2. Methodology

The authors propose a novel, passive fingerprinting approach that relies on the intrinsic statistical properties of a model's parameters rather than externally imposed signals.

Feature Extraction: The method focuses on the attention mechanism (Query $Q$ , Key $K$ , Value $V$ , and Output $O$ projection matrices). For every layer $l$ in the transformer, the authors calculate the standard deviation ( $\sigma$ ) of the parameters within each matrix.
Normalization: To ensure the method works across models of different scales (parameter counts) and architectures, the resulting sequences of standard deviations are normalized to have zero mean and unit variance. This preserves the relative pattern of the distribution across layers while removing scale-dependent variations.
Lineage Detection (Correlation):
- To compare two models, the authors compute the Pearson correlation coefficient ( $\rho$ ) between their normalized standard deviation sequences.
- Handling Different Depths: If models have a different number of layers, the authors use linear interpolation to align the shorter sequence to the length of the longer one, allowing for a meaningful comparison of the "shape" of the parameter distributions.
Scope: The method is applied to both attention matrices and Feed-Forward Network (FFN) parameters to provide a multi-faceted verification.

3. Key Contributions

Discovery of Intrinsic Signatures: The paper identifies that the layer-wise distribution of parameter standard deviations forms a unique "fingerprint" that is remarkably stable.
Robustness to Modification: Unlike watermarks, these fingerprints survive continued pre-training, fine-tuning, and even architectural upcycling (converting a dense model into a Mixture-of-Experts/MoE model).
A New Verification Framework: The authors provide a simple, computationally inexpensive method (requiring only torch.std()) to establish model lineage and detect unauthorized derivations.

4. Experimental Results

Cross-Family Differentiation: The method successfully distinguishes between different model families (e.g., Llama vs. Qwen vs. OLMoE), as each family exhibits distinct distributional patterns.
Validation of Known Lineages: The fingerprinting accurately identified known relationships, such as NVIDIA’s Llama-3.1-Nemotron being derived from Meta’s Llama-3.1, and community fine-tuned versions of Qwen2.5-7B.
The Pangu-Qwen Case Study: The most significant finding is the detection of a potential copyright violation. The authors found that Huawei’s Pangu Pro MoE model exhibits an extraordinary correlation with Qwen-2.5 14B (overall average correlation of 0.927).
- This similarity persists across all attention matrices ( $Q, K, V, O$ ) and FFN projections.
- The results suggest that Pangu Pro MoE was likely created via upcycling the Qwen model rather than being trained from scratch, despite technical reports claiming extensive independent training on 13 trillion tokens.

5. Significance and Implications

IP Protection: This research provides a technical tool for companies to defend their multi-million dollar investments against model theft and "plagiarism" disguised as independent development.
Scientific Integrity: By exposing the discrepancy between technical claims (training from scratch) and actual model origins (upcycling), the paper highlights the need for greater transparency and accountability in AI research and corporate reporting.
Industry Deterrence: The ability to detect "stolen" models even after massive continued training serves as a deterrent against the unauthorized use of foundation models in the competitive AI landscape.