💬 NLP

Which Models Are Our Models Built On? Auditing Invisible Dependencies in Modern LLMs

This paper introduces ModSleuth, an agentic system that recursively reconstructs and formalizes the complex, fragmented dependency graphs of modern LLMs from public artifacts, revealing critical insights into license obligations, training-evaluation coupling, and documentation inconsistencies.

Original authors: Sanjay Adhikesaven, Haoxiang Sun, Sewon Min

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sanjay Adhikesaven, Haoxiang Sun, Sewon Min

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you buy a fancy, pre-made sandwich from a high-end deli. The menu says, "Made with fresh ingredients." But you have no idea where the bread came from, who baked it, or if the cheese was processed by a machine that was previously used to make something else. In the world of Artificial Intelligence, specifically Large Language Models (LLMs), this is exactly the problem.

Modern AI models aren't just built from scratch using raw human data anymore. They are like culinary masterpieces built entirely from other chefs' recipes, pre-chopped vegetables, and sauces made by other robots.

Here is the simple breakdown of what this paper is about:

The Problem: The "Invisible Ingredient" Soup

In the past, building an AI was like baking a cake: you gathered flour, eggs, and sugar (raw data) and baked it. Today, it's more like assembling a complex meal where the flour itself was ground by a robot, the eggs were sorted by a different robot, and the recipe was written by a third robot that learned from a fourth.

The authors call this a "recursive dependency."

The Issue: If Model A uses data generated by Model B, and Model B was trained on data filtered by Model C, the chain gets incredibly long and messy.
The Confusion: These chains are hidden. A paper might say "we used a dataset," but it doesn't mention that the dataset was created by a specific version of a model from a different company. It's like a sandwich menu saying "ham" without telling you the ham came from a farm that was inspected by a specific, unlisted agency.
The Risk: If the original "ingredient" (Model C) had a strict rule like "Do not use my output to train other robots," that rule might get lost as the ingredient passes through Model B to Model A. This creates legal nightmares, biased results, or "contaminated" data that ruins the final product.

The Solution: ModSleuth (The AI Detective)

The authors built a tool called ModSleuth. Think of it as a super-powered detective or a supply-chain auditor for AI.

Instead of just reading the final menu (the model's paper), ModSleuth goes into the kitchen, checks the receipts, reads the supplier logs, and traces every single ingredient back to its source.

How it works (The Detective's Toolkit):

It doesn't just read; it investigates: It uses an AI agent (a digital detective) to read technical reports, code, and model cards.
It solves the "Name Game": Sometimes a model is called "Olmo 3" in a paper, but "Olmo-3-Think" in the code. ModSleuth is smart enough to realize these are the same thing, just wearing different hats.
It draws a Map: It builds a giant, visual map (a dependency graph) showing exactly how Model A connects to Model B, which connects to Model C, and so on.

What They Found (The Shocking Truth)

The authors tested ModSleuth on four major AI models. Here is what the detective found that humans missed:

The "Hidden Hand" Effect: They found dependencies that went 8 steps deep. For example, one model's training data was filtered by a classifier, which was trained on data generated by a model, which was trained on data from a completely different company.
The "Double Agent" Benchmarks: They found cases where a model was being tested on a specific math problem set, but that same math problem set was also used to train the model. It's like a student taking a test that they were secretly given the answers to during their study session.
The License Loophole: They found that some models were using data generated by models with strict "No-Reuse" licenses. Because the chain was so long, the final model creators didn't realize they were breaking the rules.
The "Ghost" Dependencies: Sometimes, the paper said one thing, but the code said another. The paper might claim "we used human data," but the code revealed they actually used data generated by a robot.

The Big Takeaway

The paper argues that we can no longer trust the "menus" (model cards and papers) alone. The ecosystem of AI is too complex and interconnected for humans to trace manually.

ModSleuth proves that:

AI is a giant web: Almost every model is connected to many others in ways we didn't know.
Transparency is broken: Current documentation is too flat and simple to capture these deep, multi-layered connections.
We need a new standard: Just like food labels now list allergens and origins, AI models need a "dependency map" that shows exactly where their ingredients came from, all the way back to the source.

In short, the paper says: "We built a detective to find the hidden ingredients in our AI sandwiches, and we discovered the kitchen is much messier and more interconnected than anyone realized."

Technical Summary: ModSleuth – Auditing Invisible Dependencies in Modern LLMs

Problem Statement

Modern Large Language Model (LLM) development has evolved into a deeply recursive ecosystem where models increasingly rely on other models for data generation, filtering, rewriting, evaluation, and preference learning. Unlike historical pipelines that relied primarily on raw human data, current workflows involve complex, multi-stage dependencies where upstream artifacts (e.g., reward models, judges, synthetic data generators) shape downstream model weights and development decisions.

The core problem is a lack of transparency in these dependency structures. Dependencies are often:

Fragmented: Information is scattered across technical reports, model cards, code repositories, and dataset cards.
Recursive: A model may depend on an upstream artifact whose own dependencies are documented only in separate, unrelated releases.
Ambiguous: It is difficult to distinguish between direct dependencies (affecting model weights) and indirect dependencies (influencing development decisions), or to reconcile artifact identities across inconsistent naming conventions and versions.

This opacity leads to concrete risks, including silent propagation of license restrictions, data contamination cascading through multi-hop paths, and circular evaluation biases where judge models share ancestry with the systems they evaluate. Existing auditing methods (e.g., ecosystem mapping, weight-based ancestry inference) are insufficient as they often focus on narrow notions of lineage or fail to capture non-weight-shaping influences like filtering and evaluation.

Methodology: ModSleuth

The authors introduce ModSleuth, an agentic system designed to recursively reconstruct LLM dependency graphs from public artifacts with source-grounded evidence. The system moves beyond simple information extraction, which modern agents can already perform, to address the semantic and representational challenges of dependency tracing.

Core Design Principles

Formalization of Dependency:
- Direct Dependencies: Upstream artifacts that materially affect the target model's training or weights (e.g., initialization models, synthetic data generators, OCR systems, filtering models).
- Indirect Dependencies: Upstream artifacts that do not enter training but substantially influence development decisions (e.g., evaluation models, ablation studies, methodological recipes).
- Exclusions: Artifacts that neither affect training nor materially influence development (e.g., baseline comparisons, vague inspiration) are excluded.
Representation via Operations:
Instead of fixed dependency labels, relationships are represented as operations—structured groups of edges describing a single pipeline event (e.g., generation, filtering, rewriting). Each edge includes a free-form natural language description, a coarse dependency type, and supporting source excerpts.
Identity Resolution via Lattice:
To handle inconsistent references (e.g., "Olmo 3 32B" vs. specific checkpoint IDs), ModSleuth uses an identity lattice. Artifacts are organized into families with root nodes for underspecified references, intermediate nodes for partial resolution, and canonical leaves anchored to concrete URLs. This preserves uncertainty without forcing premature resolution.

The ModSleuth Pipeline

The system operates in three phases, utilizing Claude Code as its underlying agentic engine:

Source Gathering: Collects official public artifacts (technical reports, model/dataset cards, code repos, release blogs) for a target LLM and organizes them into topical batches.
Entity Discovery and Resolution: Extracts mentions of dependent models and datasets, mapping them to the identity lattice. For Hugging Face artifacts, it uses deterministic metadata retrieval to validate canonical URLs and follow derivation relationships.
Dependency Construction and Reconciliation: Re-reads source batches against the resolved lattice to construct evidence-grounded edges. It merges claims describing the same relationship across sources, flagging inconsistencies for human review.

Recursive Expansion: The system recursively applies this pipeline to discovered upstream artifacts, allowing for the tracing of multi-hop dependencies (e.g., a target model $\to$ OCR model $\to$ OCR model's training data). Users can configure the search strategy (BFS, DFS, or beam search) to control coverage.

Key Contributions

ModSleuth System: A novel agentic framework for recursively reconstructing evidence-grounded dependency graphs from heterogeneous public sources.
Formalization of Recursive Tracing: A structured approach to distinguishing direct vs. indirect dependencies and representing heterogeneous pipeline roles through operation-centered relationships.
Large-Scale Dependency Graphs: The construction of dependency graphs for four public-artifact-rich LLM releases (Olmo 3, Nemotron 3, DR Tulu, SmolLM3), recovering 1,060 source-verified dependencies and constructing graphs with up to 8 hops of depth.
Audit Workflow: A query-driven approach where audit questions are executed as graph queries, surfacing risk patterns that are invisible in individual documents.

Results and Findings

Applied to four diverse LLM releases, ModSleuth significantly outperformed baseline LLMs and agents in recovering verified dependencies.

Performance: ModSleuth (unbounded scope) recovered 1,060 verified relationships, more than 3x the strongest baseline (GPT-5.5 Pro at 314). Even in the conservative depth-1 scope, it recovered 484 relationships, exceeding baselines by 54%.
Graph Scale: The merged graph contains 2,526 artifact nodes and 9,112 edges.
Dependency Composition:
- Direct vs. Indirect: 72% of verified edges are direct dependencies (affecting weights/data), while 28% are indirect (influencing development).
- Data vs. Weight Lineage: Upstream models shape downstream systems primarily through data operations (generation, filtering, transformation) rather than weight lineage.
- External vs. Internal: 75–82% of verified dependencies are external to the target organization, with Qwen, Llama, and OpenAI being the most dominant upstream families.

Qualitative Findings

The reconstructed graphs revealed several critical issues:

Underreporting and Inconsistencies: Discrepancies between papers, code, and model cards (e.g., DR Tulu's paper omitted the Claude Sonnet 3.7 dependency found in its ScholarQA implementation).
License-Relevant Paths: Multi-hop chains where license restrictions (e.g., Llama Community License) propagate through synthetic data pipelines (e.g., SmolLM3's FineMath data traces back to Llama-generated annotations).
Training-Evaluation Coupling: Benchmarks (e.g., IFEval, GSM8K, SWE-Bench) appearing simultaneously as training artifacts and evaluation targets, creating structural circularity.
Model-Mediated Selection: Upstream models (judges, filters) deciding which examples are created or retained, concentrating influence in a few families (Qwen, DeepSeek, Llama).
Code-Level Provenance: Critical dependencies often hidden in training scripts and YAML configurations rather than high-level documentation.

Significance and Limitations

Significance:
The paper argues that the lack of transparency in LLM dependencies is structural, not incidental. ModSleuth demonstrates that modern LLM ecosystems are far more interconnected and recursively dependent than previously recognized. The work provides a foundation for auditing, understanding, and governing these ecosystems by transforming fragmented public evidence into structured, queryable graphs. It highlights that current disclosure mechanisms (model cards, datasheets) are fundamentally too flat to capture recursive, multi-stage dependencies.

Limitations:

Lower Bound: ModSleuth identifies declared dependencies from public artifacts. It cannot recover undocumented, proprietary, or internal dependencies. Thus, the resulting graphs represent an evidence-grounded lower bound on the true dependency structure.
Ground Truth: Complete ground-truth dependency graphs do not exist, making absolute recall difficult to measure. Evaluation relies on post-hoc verification of reported evidence.
Agent Bias: Both the extraction and verification pipelines rely on Claude Code, potentially introducing shared modeling biases.
Scope: The analysis is restricted to official sources; unverified third-party claims are excluded.

The authors conclude that as LLM pipelines become increasingly recursive, transparency efforts must evolve beyond flat documentation toward schemas that explicitly represent dependency structure, evidence, and role semantics.