💻 computer science

Deforking the World of Code: A Project-Provenance Map that Recovers Cross-Forge Fork Families that Platform Graphs Cannot See

This paper introduces a curated "deforking" map for the World of Code that reconstructs cross-forge project families by collapsing shared git histories into unified clusters, thereby correcting popularity inflation and revealing thousands of fork relationships—including multi-forge families and non-GitHub roots—that are invisible to platform-specific graphs.

Original authors: Audris Mockus

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Audris Mockus

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the entire history of software development as a massive, chaotic library. In this library, there are millions of books (repositories). But here's the catch: many of these books are just photocopies of the same original story.

In the world of coding, this is called forking. A developer takes an existing project, copies it, and starts their own version. They might tweak a few lines here or there, but the core history is identical.

The problem the paper addresses is that if you try to count how "popular" a piece of code is by just counting every single book in the library, you get a wildly inflated number. If one popular story has been copied 10,000 times, it looks like 10,000 different stories are being read, when really, it's just one story being read in 10,000 different places.

This paper introduces a new map (a tool) that cleans up this library. It groups all the photocopies back together with their original source, so researchers can see the real story, not the noise of the copies.

Here is how they did it, using simple analogies:

1. The "Shared Page" Detective

The authors realized that in the digital world, you can't easily fake a history. If two books share the exact same page (a specific "commit" or change in code), they must be related.

The Old Way: They tried to link every book that shared a page. But this was like saying, "If two books both have a page that says 'Copyright 2024', they are the same story." That's wrong! Many unrelated books have the same copyright page. This caused the map to glue together completely different stories into one giant, messy blob.
The New Way: They built a smarter map. They looked for many shared pages, not just one. If two books share a whole chapter, they are definitely related.

2. The "Size Limit" Filter (The Cap)

Even with the smarter map, some giant, boring pages (like standard license agreements or empty starter templates) were still acting as bridges, connecting unrelated stories.

The Fix: The authors put a size limit on these bridges. If a shared page appears in more than 250 books, they assume it's just a generic template (like a standard "Terms of Service" page) and ignore it.
The Result: This broke the giant, messy blobs apart. Suddenly, the map showed distinct families of stories instead of one giant, confusing super-cluster. It didn't break up real families; it just removed the glue that was sticking unrelated things together.

3. The "Real History" Check

The authors worried that by cutting these giant blobs, they might have accidentally chopped up a real, complex story that naturally had many parts (like a story that was translated into many languages and then recombined).

The Test: They looked at the biggest remaining group on their map. They found it wasn't a mistake; it was a real, complex story where one big project had genuinely absorbed parts of other famous projects (like a major operating system incorporating code from a web browser).
The Decision: Because this "residual" group was made of real, deep history and not just cheap glue, they decided not to cut it further. They left it intact because it represents a true, complex relationship in the software world.

4. Checking Against the "Official List"

To make sure their map was accurate, they compared it against GitHub's official "Fork List" (a list where users manually click a "Fork" button).

The Match: When they looked at projects that existed on both their map and GitHub's list, they matched up 99% of the time.
The Surprise: Their map found things GitHub's list missed!
- Cross-Forge Families: They found families of projects that started on GitHub but were copied to GitLab, Bitbucket, and other sites. GitHub's list only sees the GitHub side; this map sees the whole family tree across the internet.
- Detached Forks: They found projects that started as copies but then completely rewrote their history, so they are no longer connected to the original. The map correctly identified these as separate entities, whereas the official list might still think they are connected.

5. Why This Matters

Before this map, if you wanted to know how many different projects a single programmer worked on, you might get a fake number like "5,000 projects" just because they worked on one popular project that had 5,000 copies.

The Correction: This map fixes that. It tells you the programmer actually worked on 5 distinct projects, not 5,000.
The Outcome: It provides a clean, accurate view of the software world, separating the original stories from the photocopies, and even spotting stories that span across different websites.

In short: The authors built a tool that untangles the messy web of copied code. They used a "size limit" to stop unrelated projects from sticking together, verified their work against official records, and discovered that the software world is even more interconnected across different websites than we previously knew. They released this map for anyone to use, ensuring that future studies of software history aren't fooled by the sheer volume of copies.

Technical Summary: Deforking the World of Code

Problem Statement
Mining software repositories at a global scale is fundamentally compromised by forking. When a single upstream history is copied into thousands of derived repositories, standard metrics based on raw repository counts (such as author spread, project size, and dependency fan-out) become inflated by orders of magnitude. A naive approach to "deforking"—merging repositories that share git history—fails catastrophically because it over-merges unrelated software. This occurs when trivially shared commit groups (e.g., empty commits, identical initial scaffolding, or boilerplate LICENSE/CI files) act as "hubs" that bridge disjoint fork families into giant, meaningless clusters. This phenomenon mirrors the over-merging issues found in author-identity disambiguation.

Methodology
The paper presents a curated deforking map for the World of Code (WoC) V2604 collection, extending a previous shared-commit approach to a dataset of 5.87 billion commits and 116 million repositories. The methodology involves three primary stages:

Hub-Node Star Encoding: Instead of connecting repositories sharing a commit as a quadratic clique ( $O(N^2)$ ), the authors materialize the shared-commit relation as a bipartite graph. For each distinct set of repositories sharing an identical commit, a single "hub" node is created, connected via edges to all participating repositories. This linearizes the encoding to $O(N)$ .
Parallel Louvain Clustering: The resulting bipartite graph is clustered using Parallel Louvain (PLM) to identify communities. Each community's canonical representative (excluding the hub nodes) becomes the deforked project $P$ .
Structural Capping: To prevent the over-merging caused by boilerplate hubs, the authors introduce a deterministic size cap ( $C$ ). During graph construction, any shared-commit group exceeding size $C$ is discarded before node assignment. This removes "giant hubs" bridging unrelated families while preserving genuine fork families connected by many smaller shared-commit groups. The authors sweep $C$ values (specifically $C=250$ and $C=500$ ) to determine the optimal threshold.

Key Findings and Contributions

The Mega-Cluster Problem and the Cap (Exp. D1): Without capping, the largest community absorbs 861,948 repositories. Applying a cap of $C=250$ reduces the largest cluster to 183,654 repositories (a 78.7% reduction) and collapses the tail of massive clusters (from 13 clusters $\ge 10^5$ members down to 1). Crucially, this cap does not destroy genuine fork families; the count of multi-repository families remains stable (increasing slightly from 14.34M to 14.35M), indicating the cap severs bridges between families rather than shattering the families themselves.
Residual Analysis (Exp. D2): A structural diagnostic using betweenness centrality on the largest capped cluster reveals that the remaining "residual" is not a clustering artifact but genuine shared history. The high-betweenness nodes are aggregator repositories (e.g., qtwebengine-chromium, Debian's mono-repos) that have genuinely vendored upstream histories. Unlike author-identity graphs where a structural cut dissolves the mega-cluster, cutting the top betweenness nodes in the project graph only shrinks the cluster by ~20%, confirming the residual is a dense web of true shared history that should not be surgically removed.
Impact on Developer Signals (Exp. D3): The cap significantly restores the "spread" signal (the number of distinct projects an author touches). Uncapped deforking collapses the mean spread from 89.6 to 4.4, effectively erasing the signal for prolific multi-project authors. The $C=250$ cap restores the mean spread to 38.2 and recovers the count of authors with $>1,000$ projects from 15 back to 2,213, while still collapsing genuine forks.
External Validation (Exp. D4): The map is validated against GitHub's declared fork graph (reconstructed from GHArchive ForkEvents). Conditional on both endpoints existing in WoC, there is a 99.01% edge agreement between the commit-based deforking and GitHub's declared forks. The remaining 0.99% divergence consists of forks that have completely diverged or never shared a commit, which is the correct behavior for a commit-based method. The analysis also identifies 11.04% of declared fork parents that WoC has not yet ingested, serving as a completeness metric.
Cross-Forge and Detached Forks (Exp. D5): Because the map relies on shared commits rather than platform metadata, it identifies fork families invisible to single-platform graphs.
- Cross-Forge Families: 5.41% of fork families span multiple forges (e.g., GitHub, GitLab, Bitbucket).
- Non-GitHub Roots: 1.51% of fork families have a canonical root not hosted on GitHub.
- Detached Forks: The analysis identifies 455,550 "hard-detached" edges where a declared fork has no shared commit history with its parent in WoC. Of these, 240,441 represent genuine independent origins that have grown their own downstream trees.
Refreshed Exclusion List: The authors release an updated fork-exclusion list containing 134.1 million children (3.4× larger than the previous GHTorrent-era list), refined to exclude only those forks that are not detached.

Significance and Claims
The paper claims that this deforking map provides a necessary prerequisite construction for any global-scale analysis of software repositories, correcting the inflation of popularity and spread metrics caused by fork multiplicity.

The authors emphasize that their approach is complementary to platform metadata. While platform graphs (like GitHub's) only record forks created via UI actions within a specific window, this commit-based approach recovers fork relationships that span different forges and survive history detachment. The map captures "WoC-only" fork families that platform graphs cannot represent.

The authors explicitly state that the uncapped map (currently distributed by WoC) should remain intact regarding the "residual" mega-cluster, as the structural diagnostic proves it represents genuine vendored history rather than an artifact. However, they recommend the capped map ( $C=250$ ) for future adoption to eliminate boilerplate over-merging, noting that the uncapped version currently in circulation suffers from the over-merge of unrelated projects bridged by trivial shared commits.

All artifacts, including the deforking maps, capped variants, fork-exclusion lists, and replication code, are released as a self-contained, independently hosted package keyed to the WoC V2604 collection.