🤖 AI

TokenPrint: A Calibrated Token-Space Fingerprint for Language-Model Provenance

TokenPrint introduces a training-free, calibrated token-space fingerprint that leverages top- $k$ vocabulary projections from knowledge probes to accurately identify language model provenance, lineage, and shared training data across diverse model families and quantization levels.

Original authors: Yuqi Wu, Shengming Zhao, Jie Chen

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yuqi Wu, Shengming Zhao, Jie Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out who a mysterious person is, but they never show their ID, and their parents won't tell you their family name. You only have their voice. If you listen closely, you might notice they use the same slang as their siblings, finish each other's sentences in a specific way, or have a unique rhythm that their cousins share. In the world of artificial intelligence, these "people" are Large Language Models (LLMs)—computer programs that write, chat, and solve problems. For a long time, if a company released a new model, they might hide exactly which older model they started with or what specific books and websites they were trained on. This is a problem because if a model is built on stolen data or trained to be harmful, we need to know its "family tree" to hold the right people accountable. Scientists have tried to solve this by looking at the model's internal code (its "DNA"), but what if that code is hidden? This paper asks a clever question: Can we identify a model's family just by listening to what it says?

The researchers, Yuqi Wu, Shengming Zhao, and Jie Chen from Fudan University, introduced a new tool called TokenPrint. Think of it as a "voiceprint" for AI. Instead of needing to see the model's secret internal code, they simply ask the model 250 specific questions—like "What is the capital of Canada?" or "Write a line of code for a loop"—and then look at the top few words the model chooses to answer. They call these top choices the model's "fingerprint."

Here is the magic trick: When two models are related (for example, one is a "fine-tuned" version of the other, or they were both trained on the exact same pile of data), they tend to pick the same top words for these questions, even if they are different sizes or made by different companies. The researchers measured this similarity using a math tool called the Jaccard overlap, which basically asks, "How many of the top words do these two models share?"

The paper found some fascinating things:

The Family Resemblance is Real: Models that share a "parent" or a training dataset have a much higher similarity score (around 0.35 to 0.48) than models that are totally unrelated (around 0.17). It's like how a child's voice might sound more like their parent's than a stranger's, even if they aren't identical.
It Happens Early: This "voiceprint" starts forming incredibly fast. The researchers saw these similarities appear within the first 1% of the model's training time, long before the model was actually smart enough to answer the questions correctly. This suggests the fingerprint comes from the data the model ate, not just how smart it became.
It Works Even When Hidden: The method works even if the models use different "vocabularies" (different lists of words) or different computer architectures. It's like recognizing a family member even if they are speaking a slightly different dialect.
It's a Detective Tool, Not a Magic Wand: The fingerprint is great at narrowing down the list of suspects. For example, when they tried to find the "parent" of five specific new models (called R1 distillations), the fingerprint correctly identified the true parent as one of the top two guesses every time. However, it sometimes couldn't tell the difference between a parent and a very close cousin (like a sibling model), meaning it points you in the right direction but doesn't always give a single, perfect answer.

The researchers also checked if this fingerprint survives when the model is shrunk down to save space (a process called quantization). They found that even when the model was compressed to a very small size (int4 or int8), the fingerprint remained strong, with a similarity score of 0.82 to 0.92 compared to the original. This means the "voice" doesn't change much even when the model is squished.

In short, TokenPrint suggests that every language model leaves a unique, persistent trail of "word choices" that reveals its training history. It's a training-free, lightweight way to trace a model's lineage, helping us understand who made what and where the data came from, without needing to peek inside the model's secret code. While it can't always name the exact parent with 100% certainty, it's a powerful new magnifying glass for tracking the family trees of AI.

Technical Summary: TokenPrint

Problem Statement
The rapid release of open-weight language models has outpaced the ability to audit their provenance. Current governance mechanisms face significant challenges: model cards often omit the specific base checkpoint used for fine-tuning, training data disclosures are frequently incomplete, and models nominally developed independently may share substantial portions of their training corpora. Existing forensic methods are limited in scope: weight-space approaches (e.g., HuRef, Model Tree Heritage Recovery) require parameter access and assume parameter inheritance, while behavioral phylogenies (e.g., PhyloLM) often produce uncalibrated structures that reveal clustering but lack calibrated expectations for similarity levels arising from shared bases, overlapping data, or mere capability convergence. There is a need for a method that can infer model lineage and training history without requiring access to model weights, applicable even when models share no parameters.

Methodology
The authors introduce TokenPrint, a training-free fingerprint based on the top- $k$ vocabulary projections of late hidden states elicited by a fixed set of 250 knowledge probes.

Fingerprint Extraction: For a causal language model with $L$ layers, the method uses 250 fixed prompts (probes) without chat templates. It records the hidden state $h_\ell(p)$ at specific depths ( $\ell \in \{L/4, L/2, 3L/4, L-2\}$ ) and projects them through the unembedding matrix $W_U$ to obtain logits. The top-100 tokens are decoded into strings, filtered for "glitch" tokens (based on unembedding norms), and stripped of whitespace. The fingerprint is the mapping from each probe to its ranked list of decoded token strings.
Similarity Metric: Similarity between two models $a$ and $b$ at depth $\ell$ is defined as the mean Jaccard overlap of their top- $k$ decoded string sets across all probes:
$S_\ell(a, b) = \frac{1}{|P|} \sum_{p \in P} \frac{|F_{a,\ell}(p) \cap F_{b,\ell}(p)|}{|F_{a,\ell}(p) \cup F_{b,\ell}(p)|}$
Internal depths are matched by fractional position to accommodate models of different sizes.
Calibration and Null Estimation: To account for shared lexical structures and tokenizer effects independent of probe content, the authors estimate a pair-specific "mismatched-probe null" ( $\tilde{S}$ ). This is calculated by pairing model $a$ 's response to probe $i$ with model $b$ 's response to a different probe $j \neq i$ . The final metric is the excess similarity ( $S - \tilde{S}$ ), which isolates the signal locked to probe content.
Evaluation Pool: The method is evaluated on a calibration pool of 32 open-weight models (0.6B–32B parameters) from nine developer families, including documented relationships such as Pythia models (identical data, no shared weights), DeepSeek-R1 distillations (shared base checkpoints), and successive generations. An external "witness pool" includes models trained on the same corpus (The Pile) by different organizations with different tokenizers and architectures.

Key Results

Calibrated Similarity Ladder: The fingerprint similarity broadly follows documented model relatedness.
- Identical Data (No Shared Weights): Pythia models trained on the same data in the same order show the highest excess similarity (0.35).
- Shared-Base Fine-tunes: Models inheriting a base checkpoint show high similarity (0.33).
- Same Developer/Lineage: Models from the same developer or lineage group show intermediate similarity (0.25–0.28).
- Unrelated Models: Models with no documented relationship show the lowest similarity (0.17).
- This signal persists across different organizations, tokenizer families (e.g., Cerebras-GPT vs. Pythia), and architecture classes (e.g., RWKV vs. Transformers).
Early Emergence: The similarity signal emerges within the first 1% of training (e.g., at step 1000 for Pythia models), well before measurable factual competence (probe accuracy) is achieved. This suggests the signal stems from shared training data distributions rather than just the convergence of task capabilities.
Lineage Retrieval: As a nearest-neighbor retrieval method, TokenPrint successfully ranks the exact documented base checkpoint within the top two candidates for all five DeepSeek-R1 distillations (Mean Rank 1.8, MRR 0.60). Notably, it identifies specialized bases (e.g., a math-specialized base) that coarse metadata (tokenizer lineage and parameter count) fails to distinguish. However, the method often identifies a "lineage neighborhood" rather than a unique parent, as closely related alternatives (e.g., same-base Instruct variants) often rank highly.
Depth and Robustness:
- Depth: Discrimination strength increases toward the output distribution, with AUC rising from 0.72 at quarter depth to 0.90 at the final output. Using only the top 5 output tokens retains an AUC of 0.87.
- Quantization: The fingerprint remains stable under quantization, showing Jaccard similarity of 0.92 under int8 and 0.82–0.85 under int4 compared to the bf16 baseline.
- Capability Control: Regression analysis confirms that the observed lineage-group structure persists even after controlling for capability gaps and parameter scale.

Significance and Claims
The paper positions TokenPrint as a practical tool for lineage retrieval and model-provenance analysis that operates without weight access. Its primary significance lies in:

Training-Free Operation: It requires only forward passes, making it applicable to black-box or restricted-access models where weight-space methods fail.
Calibrated Interpretation: By establishing a "similarity ladder" against documented relationships, the method moves beyond uncalibrated clustering to provide reference ranges for what levels of similarity imply specific types of relationships (e.g., shared data vs. shared base).
Data-Driven Signal: The persistence of the signal across independent organizations and architectures, and its emergence before capability convergence, suggests that token-space fingerprints capture persistent structural artifacts of training data distributions.

Limitations and Scope
The authors explicitly state that the method does not identify the mechanism of a relationship (e.g., distinguishing weight inheritance from overlapping corpora) based on a single pairwise score alone. Instead, it provides a global similarity structure to narrow candidate sets and generate hypotheses for further targeted analysis. The method is best suited for identifying which model relationships warrant deeper investigation rather than serving as standalone legal proof of a specific training history. The authors also note that cross-tokenizer comparisons may underestimate similarity due to different segmentation of semantically equivalent strings.

More like this