🤖 machine learning

Training Leaves Traces: Centered Residual Signatures for Language Model Lineage Verification

This paper introduces a data-free, white-box method called Centered Residual Signatures that verifies the lineage of open-weight language models by analyzing unique structural traces in residual blocks, enabling accurate distinction between related and unrelated checkpoints regardless of fine-tuning, merging, or quantization.

Original authors: Aman Singh Thakur, Rayan Khoury

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Aman Singh Thakur, Rayan Khoury

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern digital landscape, artificial intelligence models are rarely static, solitary creations. They are living entities that evolve through a complex supply chain. A base model, trained on vast amounts of text, is often passed from one developer to another. Along the way, it is tweaked to perform specific tasks, compressed to run on smaller devices, or combined with other models to create something new. These operations change the model's internal numbers, its weights, but they rarely leave a clear paper trail. Once a model is released under a new name, it becomes difficult to tell if it is a genuine descendant of a famous original or if it is merely a lookalike that was built from scratch to behave similarly. This lack of transparency creates a blind spot in the ecosystem of open-source intelligence, making it hard to verify origins or protect intellectual property.

The core challenge lies in distinguishing between two very different scenarios. In the first, a model is a true child of another, inheriting its specific mathematical structure through a process of fine-tuning or merging. In the second, a model is an independent creation that happens to produce similar results. Traditional methods of verification, such as checking digital fingerprints or looking at how the model behaves, often fail here. Digital signatures break the moment a single number is changed, and behavioral tests cannot tell the difference between a shared history and a shared outcome. Researchers needed a way to look inside the model's brain and find a trace that survives these transformations, a signal that proves a shared lineage without needing the original training data or the ability to run the model.

A team of researchers has developed a method to solve this problem by examining the unique geometric structure left behind inside the model's layers. They focused on a specific architectural feature common to many modern language models: the residual block. In these structures, information flows through a main path and also skips over a side path, allowing the model to learn small corrections rather than starting from zero every time. The researchers discovered that when a model is trained, the mathematical operations in these side paths develop a very specific, shared alignment. It is as if the model learns to organize its internal components in a particular way to make these corrections efficiently. However, this alignment is so common that it appears in almost any trained model, making it useless for proving ancestry on its own.

The breakthrough came when the researchers realized they could isolate the part of the structure that is unique to a specific checkpoint. By mathematically removing the common, shared alignment that all trained models possess, they were left with a residual signature. This remaining piece is like a unique fingerprint that is specific to the exact history of that model's training. It captures the specific way the model's internal numbers were adjusted during its creation. The team then created a scoring system to compare these signatures between two different models. If the models share a parent, their unique signatures will align closely, even if one has been heavily modified, compressed, or merged with another model. If they are unrelated, the signatures will look like random noise when compared.

To test this idea, the researchers ran a series of rigorous experiments using models of varying sizes and complexities. They took base models and subjected them to common real-world transformations, such as fine-tuning them on new data, reducing their size through pruning, or compressing them to run faster. They also created independent models that were trained from scratch to mimic the behavior of the originals. The results were striking. The new scoring method successfully identified every single true descendant, regardless of how much the model had been altered. It distinguished them from independent models with perfect accuracy, even when those independent models were designed to behave almost identically. The method worked across different families of language models, including those used for coding and general conversation, proving that the signal is a fundamental property of how these networks learn.

The researchers also tested the method against a more aggressive scenario where an adversary tried to hide the model's origins. They attempted to shuffle the internal components of the model or rescale its numbers in a way that preserves the model's function but scrambles its appearance to traditional detection tools. While older methods failed completely under these conditions, the new signature remained stable. This is because the method does not rely on the order of the components or their absolute size, but on the specific geometric relationship between them. The researchers found that their approach was also significantly faster than the nearest competing technique, running dozens of times quicker on large models. This speed, combined with its ability to work without needing any training data or the ability to run the model, makes it a practical tool for auditing the supply chain of open-source artificial intelligence.

In a real-world demonstration, the team applied their method to a collection of public language models that were claimed to be related to a famous original. The test correctly identified the three models that were genuine descendants, assigning them high similarity scores. At the same time, it correctly flagged seven other models that shared the same architecture but were trained independently, assigning them scores near zero. This confirmed that the method can distinguish between a true lineage and a mere architectural clone. The study suggests that this passive, data-free signal is a reliable way to verify the history of open-weight models. It offers a way to see the invisible connections between models, ensuring that the lineage of artificial intelligence can be traced, verified, and understood, even when the original creators have moved on.

Technical Summary: Centered Residual Signatures for Language Model Lineage Verification

Problem Formulation

The paper addresses the challenge of data-free white-box lineage verification for open-weight language models. As models are increasingly distributed through complex supply chains involving fine-tuning, quantization, pruning, merging (e.g., LoRA), and redistribution under new names, their provenance often becomes undocumented or intentionally obscured.

The core task is defined as follows: Given two compatible model checkpoints (A and B) with the same residual architecture (same number of layers $L$ and hidden dimension $d$ ), a verifier must determine from weights alone whether they share a weight-level ancestry.

Related: Checkpoints that inherit weights from a common ancestor through post-training operations (fine-tuning, RLHF, pruning, quantization, merging).
Unrelated: Checkpoints initialized independently, even if they share architecture, training data, or behavioral outputs (e.g., a distilled student trained from scratch is unrelated to its teacher).

Existing methods fail in this setting: cryptographic hashes break with any weight change; metadata requires trust; watermarks must be inserted proactively; and behavioral similarity metrics (CKA, SVCCA) require data/forward passes and measure functional similarity rather than weight inheritance.

Methodology: Centered Residual Signatures

The proposed method leverages a structural property of trained residual networks: trace concentration in branch products.

1. Trace Concentration in Branch Products

In a residual block $x_{\ell+1} = x_\ell + F_\ell(x_\ell)$ , the residual branch $F_\ell$ typically consists of linear layers interleaved with nonlinearities. By composing the linear factors and dropping nonlinearities, one obtains a branch product matrix $M_\ell = W_K \dots W_1 \in \mathbb{R}^{d \times d}$ .
Under the Frobenius inner product, any matrix can be decomposed into an identity-aligned component and a traceless remainder:
$M_\ell = \frac{\text{tr}(M_\ell)}{d}I + E_\ell$
where $\text{tr}(E_\ell) = 0$ .
Training induces a strong identity alignment in correctly paired branch products, meaning the trace concentration score $s(M) = |\text{tr}(M)| / \|M\|_F$ becomes high. However, this property is generic; independently trained models also exhibit high trace concentration. Thus, trace concentration alone cannot establish lineage.

2. Centered Residual Signatures

To isolate ancestry, the method removes the generic identity-aligned component and focuses on the checkpoint-specific remainder $E_\ell$ (denoted as $R_\ell$ in the paper).

Signature Extraction: For each block $\ell$ , the traceless remainder is normalized to a unit vector $\phi_\ell$ .
Alignment: A similarity matrix $G_{ij} = \langle \phi^A_i, \phi^B_j \rangle$ is constructed between reference $A$ and suspect $B$ . The Hungarian algorithm is used to recover the optimal block permutation $\pi^*$ , recovering block correspondence with high accuracy (100% on canonical MLP paths).
Lineage Score: The final symmetric score $L(A, B)$ is the average cosine similarity of the centered signatures across aligned blocks:
$L(A, B) = \frac{1}{L} \sum_{\ell=1}^L \langle \phi^A_\ell, \phi^B_{\pi^*(\ell)} \rangle$
Verification: The score is calibrated against an empirical null distribution generated from independently trained models. A checkpoint is deemed "Related" if $L(A, B)$ exceeds the maximum score observed against independent models.

3. Theoretical Mechanism

The paper proposes a gradient-coupling account for the observed identity alignment. In residual networks, the skip connection forces the branch to learn a correction around an identity path. Consequently, the input and output weights of the branch ( $W_{in}$ and $W_{out}$ ) are updated via correlated gradients that accumulate along the diagonal. This coupling is specific to the training dynamics of a specific checkpoint and survives function-preserving transformations (like neuron permutation or reciprocal rescaling) but is destroyed by re-initialization.

Key Contributions

Problem Formulation: Defined data-free pairwise weight-lineage verification and introduced centered residual signatures that isolate checkpoint-specific structure from generic training artifacts.
Model-Level Score: Constructed a symmetric score with empirical-null calibration and algebraic invariance to hidden-unit permutation and reciprocal rescaling within MLP blocks.
Robust Evaluation: Demonstrated that the method separates descendants (fine-tuned, LoRA-merged, pruned, quantized) from independent and distilled models with AUROC=1.0 on controlled benchmarks. Crucially, the method remains invariant under function-preserving checkpoint laundering (permutation and rescaling), whereas standard weight-space baselines (cosine similarity, Frobenius distance) fail or lose significant margin.
Generalizability: Showed that the underlying trace-concentration signal appears across six language model families (GPT-2, BERT, LLaMA-2, Mistral, Qwen2.5, DeepSeek-R1) and extends to vision (ViT, ResNet) and speech (Whisper) architectures.

Experimental Results

Benchmarks: Evaluated on residual-MLP and 30M-parameter GPT-2 benchmarks.
- Lineage Detection: Achieved AUROC=1.0 and a high Gap-Z margin (e.g., +31.0 on GPT-2) in distinguishing related from unrelated checkpoints.
- Laundering Resistance: Under function-preserving laundering (permutation and rescaling), the proposed method maintained AUROC=1.0. In contrast, weight cosine similarity dropped to AUROC=0.80 (with a collapsed Gap-Z margin), and aligned Frobenius distance failed under strong rescaling.
- Efficiency: The method runs 76× faster than the nearest robust baseline (Re-Basin+scale) on GPT-2 (5ms vs 388ms) because it avoids iterative alignment searches, computing branch products directly.
Public Case Study: Applied to LLaMA-2 public checkpoints. The method correctly identified 3 documented descendants (Chat, Vicuna, CodeLlama) with high scores ( $L \ge 0.33$ ) and correctly classified 7 independently trained architectural clones (OpenLLaMA, Amber, Baichuan, etc.) as unrelated ( $|L| < 5 \times 10^{-5}$ ).
Distinction from Behavior: The method successfully distinguished weight ancestry from behavioral similarity. Distilled students, which mimic teacher outputs, scored near zero ( $L \approx 0.002$ ), confirming the method detects shared parameters, not just functional mimicry.

Significance and Limitations

Significance:
The paper establishes a passive, data-free provenance signal for compatible open-weight language models. It enables the auditing of model supply chains to detect undocumented ancestry, distinguishing true weight descendants from models that merely behave similarly or share architecture. The method is particularly significant for its robustness against laundering attempts that rearrange internal structures without changing model behavior.

Limitations (as stated by the authors):

White-Box Requirement: Requires access to weights; API-only models cannot be verified.
Architecture Constraints: Limited to residual architectures (e.g., Transformers, ResNets). Plain feedforward networks, RNNs, and state-space models are out of scope.
Compatibility: Requires matching depth and hidden dimensions; cross-architecture verification is not supported.
Symmetry: The score is symmetric and cannot determine the direction of descent (ancestor vs. descendant) without external metadata.
Partial Inheritance: While the score reflects the fraction of shared blocks in linear merges or layer grafts, it does not reconstruct multi-hop ancestry or family trees.
Calibration: Requires a null distribution from independently trained models of the same architecture. The effective sample size for statistical guarantees is limited by the number of independent roots, not the number of pairwise comparisons.
Robustness to Aggressive Attacks: While robust to standard laundering, heavy pruning (85% sparsity) or extensive continued pretraining can weaken the signal (though it remains detectable above the null). Orthogonal rotation of the residual stream (if function-preserving) could theoretically destroy the signal, though such architectures are not standard in deployed models.