Technical Summary: Matrix-Driven Identification and Reconstruction of LLM Weight Homology

1. Problem Statement

The paper addresses the challenge of detecting weight homology between Large Language Models (LLMs). As models grow in scale and complexity, concerns regarding unattributed reuse, replication, or "up-cycling" (e.g., fine-tuning, pruning, continual pretraining) of model weights have intensified. The core problem is formulated as a binary classification task: given two models $A$ and $B$ with parameters $\theta_A$ and $\theta_B$, determine whether they share a weight lineage (i.e., one is derived from the other) based solely on the statistical properties of their weights.

The authors define homology to include various evolutionary paths:

• Fine-tuning: Including SFT and RL.
• Continual Pretraining: Training on additional data.
• Upcycling: Expanding a smaller model into a larger one (e.g., MoE).
• Pruning: Reducing model size by removing channels or neurons.
• Transformations: Including permutations, scaling, and orthogonal/unitary transformations.

Crucially, the paper distinguishes homology (shared initialization/weights) from homogeneity (similar outputs due to training on similar data). Two models trained on the same data with different random seeds are not considered homologous.

2. Methodology: MDIR

The authors propose Matrix-Driven Identification and Reconstruction (MDIR), a white-box method that operates exclusively on model weights without requiring model inference. MDIR leverages matrix analysis, polar decomposition, and Large Deviation Theory (LDT) to achieve three goals:

1. Identify orthogonal relationships between weight matrices.
2. Detect permutations and scaling factors.
3. Provide rigorous statistical p-values for the significance of the detected relationships.

2.1 Theoretical Foundation: Invariant Transformations

The method is grounded in the observation that modern LLMs are overparameterized, and many distinct parameter configurations produce identical input-output behaviors. These configurations form an invariant space $M_{inv}(\theta)$. The authors identify a Lie group $G$ of "totally invariant" transformations (e.g., orthogonal rotations, permutations, and scalings) that preserve the model's function.

Under idealized conditions, training dynamics are confined to the quotient space $M_{arch}/G$. Consequently, the component of the parameters along the $G$-orbit (the "G-coordinates") remains constant throughout training. If two models are homologous, their relative transformation $g_1^{-1}g_2$ should be close to the identity element $e$ of the group $G$. Conversely, independently initialized models will have a relative transformation distributed randomly across $G$.

2.2 Algorithmic Steps

The MDIR process involves the following steps:

1. Orthogonal Relation Estimation:
   For a pair of weight matrices (e.g., embeddings $E$ and $E'$), the method solves an optimization problem to find the orthogonal matrix $U$ that minimizes $\|E' - EU^T\|_F$. This is equivalent to finding the orthogonal factor in the polar decomposition of $E'^T E$.

   • Handling Different Tokenizers: If models use different tokenizers, the method restricts the comparison to a set of common vocabulary tokens $C$ to estimate the transformation.

2. Permutation Identification:
   The estimated orthogonal matrix $\tilde{U}$ is analyzed to detect specific structural patterns, such as permutations. The method solves a Linear Sum Assignment Problem (using the Hungarian algorithm) to find a permutation matrix $P$ that maximizes $\text{Tr}(P \tilde{U}^T)$. This step identifies if neurons or channels have been reordered.

3. Statistical Significance (p-value Estimation):
   To determine if the observed alignment is statistically significant or a random coincidence, the authors apply Large Deviation Theory (LDT).

   • Null Hypothesis: The weights are unrelated, and the estimated orthogonal matrix is uniformly distributed over the orthogonal group $O(n)$ according to the Haar measure.
   • Metric: The trace of the product of the permutation and the orthogonal matrix, $\text{Tr}(P \tilde{U}^T)$, serves as the test statistic.
   • Calculation: Using LDT, the probability of observing a trace value as high as the one found by chance decays exponentially with the square of the trace. The p-value is estimated as $p \le n! \cdot K \exp(-c^2/2)$, where $n$ is the embedding dimension and $c$ is the observed trace. This allows for the generation of extremely small p-values (e.g., $10^{-1000}$), providing rigorous statistical evidence of homology.

4. Layer-wise Reconstruction:
   For models with different architectures (e.g., different layer counts), MDIR iterates through layers to reconstruct the correspondence map, identifying which layers in the base model correspond to which layers in the derived model.

3. Key Contributions

• SOTA Performance: MDIR achieves perfect scores (1.000) on both Accuracy and Area-Under-Curve (AUC) across different source models in the LeaFBench benchmark, outperforming existing methods like Gradient, REEF, HuRef, and PDF.
• Statistical Rigor: Unlike previous methods that output similarity scores requiring arbitrary threshold calibration, MDIR outputs true statistical p-values, eliminating the need for heuristic threshold tuning.
• Inference-Free & Low-Resource: The method requires no model inference and compares only single pairs of matrices, making it feasible to run on low-resource devices.
• Robustness to Transformations: The method is invariant to permutations, scaling, and orthogonal transformations. It remains effective even when models use different tokenizers or have undergone pruning and upcycling.
• Reconstruction Capability: MDIR can reconstruct the specific weight correspondence mappings (e.g., identifying exactly which layers were pruned or how neurons were permuted), offering interpretability that fingerprinting methods lack.

4. Experimental Results

The paper evaluates MDIR on a diverse set of models, including:

• Instruction Fine-tuning & Continual Pretraining: Successfully identified homology between models like Llama-3.1-8B and Llama-3.2-1B, and Qwen2.5 variants.
• Pruning: Correctly detected that Llama-3.2-1B and 3B are pruned versions of Llama-3.1-8B, reconstructing the specific layer mapping (e.g., identifying that Llama-3.2-1B retains specific layers from the 8B model).
• Upcycling: Identified relationships in MoE models (e.g., Qwen1.5-MoE-A2.7B derived from Qwen1.5-1.8B).
• Different Tokenizers: Demonstrated effectiveness by retraining a model with a new tokenizer (GPT-NeoX) while retaining the original weights. MDIR successfully detected homology as the new embeddings converged, even with a changed vocabulary.
• Adversarial Robustness:
  • Noise Injection: MDIR remained robust against Gaussian noise injection, retaining high detection confidence even when noise levels caused the model's functional performance (MMLU accuracy) to collapse.
  • Independent Models: Correctly identified no homology between independently developed models (e.g., DeepSeek-V3 and Kimi-K2) or models trained on the same data with different random seeds.

5. Significance and Claims

The authors claim that MDIR represents a fundamental shift in model auditing by moving from heuristic similarity metrics to statistically rigorous hypothesis testing.

• Extreme p-values: The paper notes that the resulting p-values (e.g., $10^{-104}$) are not numerical artifacts but accurate reflections of the statistical improbability of such weight alignments occurring by chance in high-dimensional spaces, driven by Large Deviation Theory.
• Accountability: The method provides a tool for fostering accountability and transparency in AI development, potentially aiding in the detection of intellectual property violations and unauthorized model replication.
• Limitations: The authors modestly acknowledge that while MDIR excels at pairwise homology detection, it does not yet solve complex phylogenetic questions (e.g., determining the exact evolutionary tree or common ancestors among three or more models).

In summary, MDIR offers a mathematically grounded, inference-free, and highly accurate framework for verifying the provenance of LLM weights, capable of detecting subtle evolutionary relationships that previous black-box or similarity-based methods miss.