Removing Noise or Introducing Bias? The Hidden Cost of MSR Filtering

Imagine the internet as a giant, chaotic library where anyone can build a bookshelf and call it a library. This is the world of Open Source Software (OSS), a massive digital playground where millions of people—from students testing out new coding ideas to professional developers building the next big app—store their projects. Researchers, who are like detectives trying to solve mysteries about how software works, love to visit this library. They dig through the shelves, looking at how many people "liked" a project (stars), how many times people changed the books (commits), and how many people helped write them. These clues help them understand the secrets of software engineering. But here's the catch: the library is so huge that it's full of empty boxes, test dummies, and half-finished doodles. To find the "real" books, researchers usually throw away anything that doesn't look popular or busy enough. They use rules like, "If a project doesn't have at least 10 stars, it's just noise, so let's toss it."

But what if those rules are throwing away the most interesting stories? What if, in our desperate attempt to clean up the library, we accidentally hide the fact that most books are actually abandoned, or that we only end up reading the ones written by the same few famous authors? This is the big question that researchers Mohit Kaushik and Jyoti Bawa are asking. They are worried that the very filters scientists use to make their data "clean" might be making their findings "dirty" by hiding the true, messy reality of how software projects actually live and die.

The Great Filter: Cleaning the Data or Hiding the Truth?

In this study, the authors decided to play a game of "What If?" with a massive pile of data. They looked at 1.57 million software repositories from a platform called SEART. Think of this dataset as a giant bucket of mixed-up LEGO bricks. Some are huge, colorful castle sets; others are tiny, single red bricks; and many are just broken pieces that no one ever finished.

Usually, researchers look at this bucket and say, "Okay, we only want the big, finished castles. Let's throw away anything with fewer than 10 stars (likes) or fewer than 10 commits (changes)." The authors tested what happens when they applied these strict rules, like turning up the volume on a filter until it gets really loud.

The Hidden Cost of "Popularity"
When the researchers applied a "popularity filter" (looking only at projects with more stars), they found something surprising. As they raised the star threshold from 10 to 1,000, the "average" project in their sample didn't just get slightly better; it got 7 times bigger. The projects became older, had more people working on them, and were much more likely to have a formal license (like a rulebook).

But here is the twist: by chasing the popular projects, they completely lost sight of the reality. In their original, unfiltered bucket, 73.42% of the projects were actually inactive or "abandoned." However, as they filtered for popularity, this number dropped. By the time they only looked at the super-popular projects (1,000+ stars), the data made it look like only 50.65% were abandoned. The filter didn't just remove the noise; it hid the fact that most projects actually fail or get left behind. It's like if you only asked the most successful people in a city about their jobs, and then concluded that "unemployment is low" because you never talked to the people who lost their jobs.

The "Activity" Trap
The authors also tested "activity filters," which keep only projects with a high number of commits (changes). This was even more extreme. When they filtered for high activity, the average project size grew by 18 times! These filters also changed the "personality" of the software. For example, projects using the C++ language were common in the lower-threshold groups but disappeared from the top 5 when the filter got strict. Meanwhile, TypeScript and Go became much more common in the filtered lists.

The study suggests that these filters aren't neutral. They act like a sieve that only lets through specific types of projects: older, huge, well-funded infrastructure projects. They push out the smaller, newer, or more experimental projects, even if those smaller projects are real and active.

The Relationship Racket
Perhaps the most playful (and dangerous) finding is about how these filters mess up the relationships between different things. Imagine you are trying to figure out if "working hard" (commits) leads to "being popular" (stars). In the real, messy world (the baseline data), these two things are only weakly connected. But when the researchers applied their filters, the connection suddenly looked super strong.

For instance, the link between "commits" and "project size" jumped from a moderate 0.466 to a very strong 0.808 in the activity-filtered group. The authors explain that this isn't because the projects actually changed; it's because the filter forced them to look that way. By only keeping the big, busy projects, the filter made it look like "big projects always have lots of commits," when in reality, the relationship is much more complicated. It's like if you only studied the tallest basketball players and concluded that "height is the only thing that matters in sports," ignoring everyone else.

The Verdict: Don't Just Throw Away the Noise

The authors conclude that while we need to clean our data, we can't just use arbitrary rules like "10 stars" or "500 commits" without thinking. These rules are like a blunt hammer: they smash away the "noise," but they also smash away the truth. They create a distorted picture where software projects seem more successful, older, and more uniform than they really are.

Instead of blindly filtering, the authors suggest researchers should use stratified sampling. Imagine taking a scoop of the LEGO bucket that has a fair mix of big castles, small houses, and broken pieces, rather than just picking out the biggest castles. They also urge scientists to rethink what counts as "noise." Maybe a project with zero stars isn't just a failed experiment; maybe it's a hidden gem that just hasn't been discovered yet.

In short, this paper warns us that in our rush to find the "perfect" data, we might be building a house of cards that looks perfect on the outside but collapses the moment we try to understand the real, messy world of software. The authors don't say we should stop filtering entirely, but they strongly suggest that we stop using these "one-size-fits-all" rules and start being more careful about what we are throwing away.