Was It Never Collected, or Rewritten Away? A Commit-Provenance Dataset Separating Ingestion Gaps from Upstream History Edits across the World of Code

Imagine you are trying to keep a perfect, complete photo album of every single moment in a massive, global art project where thousands of people are constantly painting, erasing, and repainting over each other's work.

This paper is about a new tool that helps us understand why some photos are missing from our album.

The Problem: The "Missing Photo" Mystery

For years, researchers have tried to collect every single commit (a "photo" of code changes) from GitHub into a giant database called the World of Code (WoC). But they noticed a problem: the album is incomplete. There are millions of "photos" that were supposed to be there but aren't.

Previously, researchers just assumed these missing photos were lost forever because the collection system failed to grab them. They thought, "Oh, we missed 40% of the photos; our collection is broken."

But this paper argues: Wait a minute. Maybe the photos aren't lost; maybe they were deleted on purpose.

The Two Suspects

The author, Audris Mockus, identifies two very different reasons a photo might be missing:

1. The Collection Gap (The "Missed Shot"): The camera (the collection system) blinked, or the file transfer failed, and the photo was never saved. This is a genuine error.
2. The History Rewrite (The "Deleted Draft"): The artist (the project owner) decided they didn't like a specific version of the painting. They used a "force-push" (a digital eraser) to wipe that version out of history and replace it with a new one. The photo never existed in the final version because the artist erased it.

The old way of counting lumped these two together. It said, "We are missing 46% of the photos!" This made the collection look much worse than it actually was.

The Detective Work: Two Different Viewpoints

To solve this mystery, the author used two different "cameras" to look at the same event:

• Camera A (GHArchive): This is like a security camera that records everything that was advertised the moment it happened. It doesn't care if the photo is later deleted; it just records, "At 2:00 PM, someone pushed a photo called 'Commit X'."
• Camera B (World of Code): This is the current photo album. It holds everything it has collected so far, but it doesn't keep deleted photos.

By comparing the security camera's log (what was advertised) against the current album (what is kept), the author could sort every missing photo into three neat piles:

1. Present: The photo is in the album. (53% of the time)
2. Rewritten: The photo was advertised, but the artist erased it later. The security camera saw it, but the album doesn't have it because it was deleted. (6.5% of the time)
3. Never-Ingested: The photo was advertised, the artist didn't delete it, but the album still doesn't have it. This is the true collection gap. (40% of the time)

The Big Surprise

The results were shocking. Previously, people thought the "missing" rate was nearly 47%.

• The Reality: Only 40% are truly missing (collection gaps).
• The Twist: About 6.5% of the "missing" photos were actually erased by the artists themselves.

This means that for every 15 missing photos, one of them wasn't a mistake by the collector; it was a deliberate edit by the project.

Why This Matters (In Simple Terms)

1. Honest Reporting
If you are a researcher studying software, you no longer have to say, "Our data is 47% incomplete." You can say, "Our data is 40% incomplete, and the other 7% was deleted by the creators." This makes your study much more accurate.

2. Counting Work Correctly
Imagine a worker who writes 100 lines of code, then deletes them and rewrites them as 100 new lines.

• Old Count: You only see the 100 new lines. You think they did 100 lines of work.
• New Count: This tool sees that they wrote 100, deleted them, and wrote 100 more. It realizes the "deleted" 100 lines were real work that just got hidden.
• The Result: The paper shows that if you only count what's currently visible, you are underestimating total productivity by about 10.8%. That's like thinking a factory made 90 cars when they actually made 100, because 10 were scrapped and rebuilt.

3. Avoiding Double Counting
If a developer rewrites a commit (erases it and creates a new one with a new ID), it's the same "patch" of code. Without this tool, a computer might count the old version and the new version as two separate contributions. This tool flags them as duplicates so you don't get confused.

The Bottom Line

This paper gives us a detective's magnifying glass. It separates the "broken camera" (collection gaps) from the "deleted drafts" (history rewrites).

It tells us that the World of Code is actually better at collecting data than we thought (because some of the "missing" data was never meant to be there), but it also tells us that we are underestimating how much work developers do because we can't see the work they erased.

The author has released all the data and tools so anyone can use this "detective lens" to get a clearer, more honest picture of the world of code.