🤖 AI

LLM DNA: Tracing Model Evolution via Functional Representations

This contribution introduces "LLM DNA," a mathematically grounded, training-free framework that extracts low-dimensional functional representations to trace undocumented evolutionary relationships and construct phylogenetic trees across hundreds of large language models.

Original authors: Zhaomin Wu, Haodong Zhao, Ziyang Wang, Jizhou Guo, Qian Wang, Bingsheng He

Published 2026-05-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zhaomin Wu, Haodong Zhao, Ziyang Wang, Jizhou Guo, Qian Wang, Bingsheng He

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of Large Language Models (LLMs) as a vast, chaotic library containing millions of books. Some books are original masterpieces, some are copies, some are summaries, and some are newly written versions with added chapters. The problem is that the library lacks a clear catalog. We do not always know which book was copied from which, or how one book transitioned into another. This makes it difficult to verify whether a book complies with the rules (licenses), to detect whether a dangerous idea (such as a security vulnerability) was copied from a flawed book, or to understand how the library is growing.

This article, titled "LLM DNA", proposes a solution inspired by biology: assigning each AI model its own unique genetic code.

The Core Idea: AI Genetics

Just as biological DNA tells us who your parents are and which traits you inherited, the authors aim to create a "DNA" for AI models that reveals their family history and functional personality.

They define this LLM DNA as a short, compact list of numbers (a vector) that acts like a fingerprint. If two models are related (for example, if one is a fine-tuned version of the other), their DNA should look very similar. If they are unrelated, their DNA should look very different.

How It Works (The "RepTrace" Pipeline)

The authors developed a tool called RepTrace to extract this DNA. Here is the process, simply explained:

The Test Drive: Instead of examining the model's internal code (which is often hidden or differs for every model), they treat the model as a black box. They feed it a series of random questions or prompts (like a driving test).
The Reaction: They record how the model responds.
The Translation: They convert these text responses into a "semantic map" (a mathematical representation of meaning). For example, the words "vacation" and "holiday" are different sequences of letters, but the DNA system understands that they mean the same thing.
The Compression: They use a mathematical trick (random projection) to squeeze this massive amount of information into a tiny, manageable list of numbers. This list is the DNA.

Crucially, this process is training-free. You do not need to teach the system how to find the DNA; it simply calculates it based on how the model behaves.

What They Discovered

The researchers tested this on 305 different AI models from various companies (such as Meta, Google, and Alibaba). Here is what they found:

Family Trees: When they compared the DNA of different models, they could draw a "family tree" (phylogenetic tree). This tree correctly grouped models of the same family (such as all "Llama" models) together, even though the researchers had not told the system which models belonged to which family.
Hidden Connections: The DNA revealed relationships that were not officially documented. For example, it showed that some models claiming to be based on a specific version of a family were actually closer to a different version, or that two models from different companies were surprisingly similar in their "genetics."
Evolutionary Pace: By examining the branches of the family tree, they could see which model families evolve faster than others. Some families (like Qwen) showed rapid changes, while others (like Llama) were more stable.
Robustness: The DNA remained consistent even when they changed the questions asked or used different model types to evaluate the responses. It is like a fingerprint that stays the same whether you press your finger lightly or firmly.

Why This Matters (According to the Article)

The authors suggest that this "DNA" can be used for three main applications:

License Verification: It can help verify whether a new model was illegally copied from a protected one, even if the creators do not admit it.
Security Audits: If a "parent" model has a security vulnerability (such as a backdoor), its "children" (descendants) likely have it too. The DNA tree helps auditors quickly identify these risky lineages.
Model Selection: It helps users choose the right model. If you need a model that is very stable and does not change its behavior often, choose a branch with short evolutionary steps. If you want the newest, most experimental features, choose a branch with rapid evolution.

The Conclusion

The article argues that we can mathematically prove that this "DNA" exists and faithfully captures the essence of an AI's behavior. By transforming complex, opaque AI models into simple, comparable genetic codes, we can finally begin to map, manage, and understand the exploding ecosystem of artificial intelligence.

Technical Summary: LLM-DNA: Tracking Model Evolution through Functional Representations

Problem Statement

The rapid proliferation of Large Language Models (LLMs) has created a vast, opaque landscape where millions of models exist with undocumented evolutionary relationships. Tracking how models evolve through fine-tuning, distillation, or adaptation is crucial for security audits (e.g., tracing backdoor transfers), model governance (e.g., verifying license compliance), and designing multi-agent systems. However, existing methods for defining "LLM-DNA" suffer from significant limitations:

Task Specificity: Many approaches compress models into representations optimized for specific tasks (e.g., routing or ensemble learning), failing to generalize across tasks or capture intrinsic model properties.
Fixed Sets: Some task-agnostic methods (e.g., EmbedLLM) define representations relative to a fixed set of models, meaning the representation changes as soon as new models are added, thus preventing it from representing an intrinsic property of a single model.
Structural Dependence: Provenance methods relying on token similarity or parameter correlation often fail to generalize across heterogeneous LLMs with different architectures and tokenizers.

There is a lack of a principled, formal framework to define and extract a stable, intrinsic "DNA" that captures the functional behavior and evolutionary lineage of LLMs.

Methodology

The article proposes LLM-DNA, a formal framework that mathematically defines and extracts a low-dimensional representation of an LLM's functional behavior, analogous to biological DNA.

1. Theoretical Foundation

The authors formally define an LLM as a function mapping discrete input sequences to continuous logit vectors. They define LLM-DNA as a low-dimensional vector $\tau_f$ obtained via a bi-Lipschitz mapping from the functional LLM space $\mathcal{F}$ to a DNA space $\mathcal{D}$ .

Bi-Lipschitz Condition: The mapping ensures that the distance between two DNA vectors is bounded by the functional distance between the corresponding models ( $c_1 \cdot d_H(f_1, f_2) \le d_\tau(\tau_{f_1}, \tau_{f_2}) \le c_2 \cdot d_H(f_1, f_2)$ ).
Properties: This definition mathematically guarantees two key properties:
- Inheritance: Small perturbations to a model (e.g., fine-tuning) lead to small changes in its DNA.
- Genetic Determinism: Models with similar DNA exhibit similar functional behavior.
Existence Proof: Using the Johnson–Lindenstrauss (JL) Lemma, the authors prove that such a low-dimensional embedding exists for any finite set of LLMs and preserves the geometry of the functional space.

2. Practical Extraction Pipeline: RepTrace

To extract LLM-DNA in practice without training, the authors introduce RepTrace, a scalable, training-free pipeline:

Semantic Representation: Instead of comparing raw strings, the pipeline uses a sentence embedding model to convert model responses into semantic vectors. This handles semantic equivalence (e.g., "holiday" vs. "vacation") and works with closed-source APIs.
Stochastic Functional Distance: Since evaluating the entire input space is infeasible, the method approximates the functional distance using a stochastic functional distance. This involves sampling a set of representative prompts, collecting responses, and calculating the expected Euclidean distance between the concatenated semantic embeddings of the outputs.
Random Projection: Leveraging the theoretical insight from the JL Lemma, the high-dimensional functional representation is projected into a low-dimensional DNA vector via a random Gaussian linear projection. This step is computationally efficient and requires no task-specific training.

Main Contributions

Formal Definition and Existence Proof: The article provides the first formal mathematical definition of LLM-DNA as a bi-Lipschitz representation and proves its existence using the JL Lemma.
Training-Free Extraction Framework: RepTrace is proposed as a general, scalable pipeline that extracts DNA from heterogeneous LLMs (different architectures, tokenizers, and sizes) without requiring post-training or access to internal parameters.
Empirical Validation: Extensive experiments with 305 LLMs show that LLM-DNA effectively identifies evolutionary relationships and outperforms existing baselines in relationship detection and model routing tasks.
Evolutionary Analysis: The authors construct a phylogenetic tree of LLMs based on DNA distances, revealing architectural shifts (from encoder-decoder to decoder-only), temporal progress, and previously undocumented lineage relationships.

Experimental Results

Relationship Detection: On a dataset of 305 models, LLM-DNA achieved an AUC of 0.992 for detecting correlated versus independent model pairs, significantly outperforming baselines such as Random, Greedy, and PhyloLM (AUC 0.788).
Model Routing: In a training-free model routing task, frozen LLM-DNA vectors outperformed embeddings learned by EmbedLLM (0.672 accuracy vs. 0.665), demonstrating that DNA captures intrinsic functional properties without task-specific optimization.
Stability: The DNA representation proved robust against various datasets (Mantel test correlation $r > 0.75$ ), random inputs, and variations in chat templates. It also showed that fine-tuning alters DNA values monotonically with data size, while the global structural pattern remains preserved.
Phylogenetic Insights: The constructed evolutionary tree correctly aligned with known architectural shifts and release schedules. It also identified undocumented relationships, such as grouping Microsoft's orca-2-13b and LMSYS's vicuna-7b-v1.1 within Llama-based clusters, indicating their base models despite vague documentation.

Significance and Claims

The article claims that LLM-DNA offers a principled, data-driven framework for understanding the LLM ecosystem. Its significance lies in:

Governance and Security: Providing a functional method for verifying model provenance, which is critical for enforcing license compliance and tracking the spread of security risks (e.g., backdoors) from ancestor models to descendant models.
Scalability: Unlike previous methods requiring post-training or fixed model sets, RepTrace enables immediate DNA extraction for new models, supporting the dynamic nature of the LLM landscape.
Evolutionary Insight: By treating LLMs as evolving species, the framework enables the visualization of the field's history, reveals varying evolutionary speeds within families (e.g., Qwen and Gemma evolve faster than Llama), and uncovers hidden connections that metadata alone cannot reveal.

The authors emphasize that while the current definition does not assign specific mathematical meaning to DNA subsequences, the observed stability and inheritance properties provide a solid foundation for future work in model management, risk mitigation, and targeted modification of LLM behavior.