Social Proof is in the Pudding: The (Non)-Impact of Social Proof on Software Downloads

Through two field experiments on GitHub involving the manipulation of repository stars and package download counts, the study finds that social proof metrics have no discernible impact on subsequent software downloads or developer engagement, suggesting that open-source software choices are not easily gamed by inflating these indicators.

The Gist

Imagine you walk into a massive, chaotic flea market where thousands of vendors are selling tools. You need a specific wrench, but you've never seen any of these tools before. How do you decide which one to buy?

You probably look for social proof. You ask yourself: "Who else is buying this? How many people have given this tool a thumbs-up? Is it a best-seller?"

This is exactly how software developers choose "open-source" code (the digital tools that build our apps, websites, and systems). They can't inspect every single line of code for security flaws because it takes too long. So, they rely on the "crowd's" opinion. On the platform GitHub, this opinion is measured by "Stars" (like "Likes" on Facebook) and "Downloads."

The big fear is that bad guys (hackers) could fake these numbers. They could buy fake "Stars" or use robots to fake "Downloads" to make a dangerous, virus-ridden tool look popular and trustworthy, tricking developers into using it.

The Big Question: If a hacker fakes the popularity of a software tool, will it actually trick people into downloading it?

The Experiment: "The Fake Popularity Test"

Two researchers decided to find out by running a real-life experiment, like a scientist testing a new fertilizer on a field of crops. They did this in two ways:

1. The "Star" Experiment (The Like Button Test)

• The Setup: They picked 100 brand-new, unknown software tools on GitHub.
• The Trick: For half of them, they bought fake "Stars" from a black-market website. For the other half, they asked their friends to click the "Star" button.
• The Result: They waited to see if these "popular" tools got more downloads.
• The Verdict: Nothing happened. The tools with the fake stars didn't get any more downloads than the tools with zero stars. The developers didn't fall for the trick.

2. The "Download" Experiment (The Sales Counter Test)

• The Setup: They picked thousands of software packages.
• The Trick: They wrote a computer script to "download" these packages 100 times each, making the official download counter look huge (like a store that suddenly sold 100x more items than usual).
• The Result: They waited to see if this fake sales spike made real people download the software.
• The Verdict: Again, nothing happened. The real download numbers didn't budge.

Why Didn't the Trick Work?

You might wonder, "But isn't social proof supposed to work?" (Think of how we buy the most popular ice cream flavor). The researchers explain that software developers are different from regular shoppers for a few reasons:

1. The "Stakes" are Higher: If you buy a bad ice cream, you just get a stomach ache. If a developer installs bad code, their entire website could crash, or their company could get hacked. Because the consequences are so scary, developers are much more careful. They don't just look at the "Star" count; they look at the code itself, the documentation, and the history of the project.
2. They Know the Game: Developers know that "Stars" can be bought cheaply. It's like knowing that a restaurant can pay people to write fake 5-star reviews. Because they know the signal is "noisy" (unreliable), they ignore it.
3. The "New Kid" Factor: The researchers tested this on new tools, where you'd expect social proof to matter most. Even then, the fake popularity didn't work.

The Takeaway: "The Absence of Evidence is Not Evidence of Absence"

The researchers conclude that faking a little bit of popularity doesn't work to trick developers.

However, they add a very important warning: Just because the trick didn't work in their small experiment doesn't mean it's impossible.

• Imagine if the bad guys didn't just buy 50 stars, but 50,000 stars.
• Imagine if they did this not just once, but kept it up for years.
• Imagine if the "decision maker" wasn't a human developer, but an AI robot that just picks the most popular thing without thinking.

In simple terms:
The study is like testing if a single fake "Sold Out" sign on a store window makes people buy a product. It didn't work. But if the bad guys put up a million fake signs and flooded the street with actors, they might eventually succeed.

The Bottom Line:
For now, developers are smart enough to see through small lies. But as technology changes (like AI making decisions for us), the rules might change, and the "fake popularity" trick could become a much bigger danger. The researchers are saying, "Don't get complacent; keep watching the signs."

Technical Summary

Here is a detailed technical summary of the paper "Social Proof is in the Pudding: The (Non)-Impact of Social Proof on Software Downloads."

1. Problem Statement

The paper addresses a critical security concern in the open-source software ecosystem: supply chain attacks via social proof manipulation.

• Context: Developers frequently rely on "social proof" (e.g., star counts on GitHub, download counts on PyPI) as a heuristic to select open-source packages because vetting code for security and quality is time-consuming and difficult.
• Threat: Malicious actors can "game" these metrics (e.g., buying fake stars or botting downloads) to make malicious packages appear popular and trustworthy, thereby inducing developers to install malware.
• Research Question: Does artificially inflating social proof metrics (stars or download counts) actually cause a measurable increase in subsequent organic software adoption (downloads) or repository engagement?

2. Methodology

The authors conducted two large-scale Field Experiments (Randomized Controlled Trials) on live platforms to test causality.

Experiment 1: Manipulating GitHub Stars

• Platform: GitHub (the world's largest developer platform).
• Sample: 622 new Python packages with public GitHub repositories, listed between April 24–30, 2023.
• Randomization: 100 packages were assigned to treatment groups; the rest served as controls.
• Treatment Conditions:
  1. Low Dosage: 75 packages received "stars" from the authors' personal network (median ~19 additional stars).
  2. High Dosage: 25 packages received "stars" from a combination of the network and purchased stars from a third-party vendor (Baddhi Shop), totaling ~69 additional stars (doubling the median star count).
• Outcome Measures:
  • Primary: Human downloads of the associated Python package (tracked via PyPI/Linehaul logs, filtering out bot traffic).
  • Secondary: Repository engagement metrics (forks, pull requests, issues, pushes).
• Duration: Effects were measured over a 3-month post-treatment period.

Experiment 2: Manipulating PyPI Download Counts

• Platform: PyPI (Python Package Index).
• Sample: 23,916 Python packages with at least 5 historical human downloads.
• Randomization: 4,814 packages assigned to treatment; 19,102 to control.
• Treatment: A script downloaded each treatment package 100 times to artificially inflate the official download count (increasing the median from ~20 to ~100).
• Outcome Measures: Subsequent organic human downloads and downstream GitHub repository activity.
• Duration: Measured over a 42-day post-treatment period.

Analytical Strategy

• Intent-to-Treat (ITT): Analyzed the effect of assignment to the treatment group.
• Local Average Treatment Effect (LATE): Analyzed the effect on "compliers" (packages that actually received the minimum threshold of manipulated stars/downloads).
• Statistical Approach: Focused on medians rather than means to mitigate the impact of extreme outliers (a few viral packages). Used difference-in-differences and linear trend models.

3. Key Results

The study found no statistically significant evidence that manipulating social proof metrics influences developer behavior.

• GitHub Experiment (Stars):

  • Downloads: There was no discernible difference in PyPI download counts between treated and control groups at any point up to 3 months post-treatment.
  • Engagement: No significant impact on forks, pull requests, issues, or other repository activity.
  • Manipulation Check: The treatment successfully increased star counts (median increase of 19 for low-dosage, 69 for high-dosage), confirming the manipulation worked, but it failed to translate into adoption.

• PyPI Experiment (Downloads):

  • Downloads: While the treatment successfully spiked download counts during the intervention window, there was no subsequent increase in organic downloads compared to the control group. In fact, the treated group showed a slightly flatter trend post-intervention.
  • Engagement: No spillover effect on GitHub repository activity (stars, forks, etc.).

• Robustness Checks:

  • Results held across different time snapshots and when analyzing only "compliers."
  • Heterogeneity analysis (testing if complex packages were more susceptible) yielded null results, though confidence intervals were wide.
  • Observational analysis (Granger causality) suggested human downloads predict future human downloads, but this contradicted the experimental findings, suggesting the experimental manipulation did not trigger the "herding" behavior seen in observational data.

4. Key Contributions

1. Empirical Evidence on Social Proof: This is one of the first field experiments to rigorously test the causal link between social proof metrics and software adoption. It challenges the prevailing assumption that "fake popularity" directly drives malware adoption.
2. Developer Behavior Insights: The findings suggest that software developers are less susceptible to peripheral cues (like star counts) than users in other domains (e.g., e-commerce or social media). Developers appear to rely more on "central route" processing (evaluating code, documentation, commit history) due to the high stakes of security and build failures.
3. Signal Degradation: The results support Signaling Theory, suggesting that because fake stars are cheap and widely known to exist, developers have learned to discount them as noisy signals of quality.
4. Methodological Rigor: The paper provides a blueprint for ethical field experimentation in cybersecurity, demonstrating how to manipulate live metrics without compromising user safety or data integrity.

5. Significance and Implications

• Security Implications: While the study does not rule out that massive astroturfing campaigns could eventually succeed, it suggests that current, modest levels of manipulation are ineffective. This implies that the immediate threat of malware spreading solely via inflated star/download counts may be lower than feared.
• Platform Design: The authors argue that platforms should move beyond raw popularity metrics. They recommend:
  • Displaying star velocity (to detect spikes).
  • Integrating security health scores (e.g., OpenSSF Scorecard) which are harder to fake.
  • Using cryptographic provenance (trusted publishing) to verify supply chain integrity.
• Theoretical Contribution: The study highlights the unique nature of software adoption as a "consequential technical decision." Unlike buying a product, integrating a dependency carries tangible risks (broken builds, vulnerabilities), motivating developers to look beyond surface-level popularity.
• Caveats: The authors caution that "absence of evidence is not evidence of absence." Their treatment intensity was limited by ethical constraints. They cannot rule out effects from sustained, high-volume commercial astroturfing or from automated agents (LLMs) that may lack human scrutiny.

Conclusion: The paper concludes that while social proof manipulation is a known threat, the specific mechanism of "buying stars/downloads" did not successfully induce adoption in their experiments. Developers appear to possess a degree of skepticism or alternative evaluation mechanisms that buffer them against these specific forms of social engineering.