🤖 machine learning

Gaming Consensus: Coordinated Manipulation in Crowdsourced Fact-Checking

This paper reveals that coordinated users can strategically manipulate crowdsourced fact-checking systems based on matrix factorization to fabricate synthetic consensus, leading the authors to develop and deploy specific mitigations within X's Community Notes algorithm.

Original authors: Nikil Roashan Selvam, Jay Baxter, Sophie Hilgard, Brad Miller, Keith Coleman, Ellen Vitercik, Sanmi Koyejo

Published 2026-07-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nikil Roashan Selvam, Jay Baxter, Sophie Hilgard, Brad Miller, Keith Coleman, Ellen Vitercik, Sanmi Koyejo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive, global town square where anyone can post a note next to a rumor to say, "Actually, here's the truth." This is how Community Notes works on platforms like X (formerly Twitter), Meta, and TikTok. The goal is to stop misinformation without having a single boss or editor deciding what is true.

Instead of a boss, the system uses a clever voting machine called a "Bridging Algorithm."

The Problem: The "Echo Chamber" Vote

In a normal vote, if 1,000 people who all agree with each other vote "Helpful," the note wins. But in a town square full of different viewpoints, that's dangerous. A group of bots could just pretend to be 1,000 friends to push a fake fact.

So, the system changed the rules: A note only wins if people who usually disagree with each other both say it's helpful.

If a conservative and a liberal both think a note is accurate, it gets shown.
If only conservatives think it's accurate, it stays hidden, even if there are 1,000 of them.

This is called "Bridging." It's like trying to build a bridge across a river; you need anchors on both banks to hold it up.

The Discovery: "Gaming the Bridge"

The researchers in this paper asked: "Can a bad actor trick this system?"

They found that yes, a coordinated group can "game" the bridge. They didn't hack the computer code or break a security lock. Instead, they figured out how to act like a diverse group of people.

Think of the system as a giant map where everyone is a dot.

The Old Way: To win, you needed 1,000 dots in different places on the map to all vote "Yes."
The Attack: The researchers showed that a small group of bad actors (fewer than 10 people) could first "train" their accounts to look like they live in different parts of the map.

The Two-Step Trick:

Phase 1 (The Disguise): The attackers use their fake accounts to rate other notes in very specific ways. This tricks the system into thinking, "Oh, Account A is a liberal, and Account B is a conservative." They are essentially painting themselves with different colors to look like a diverse crowd.
Phase 2 (The Trap): Once the system believes these accounts are different people with different opinions, the attackers all vote "Helpful" on a specific target note. Because the system thinks these are "different" people agreeing, it builds the bridge and shows the note to everyone.

The Shocking Twist: "No" Can Mean "Yes"

The most confusing part of their discovery is a mathematical quirk. Sometimes, to make a note look more helpful, you actually have to vote "Not Helpful."

The Analogy: Imagine a seesaw. If everyone on the left side is pushing down hard, the right side goes up. But if you add one person on the far left who pushes up (voting "Not Helpful"), it can actually tilt the whole seesaw in a way that makes the center point (the score) go up.
In simple terms: If a note is being rated by a very specific, narrow group, adding a "bad" rating from someone who looks like they belong to that same narrow group can mathematically trick the system into thinking the note is actually more universally accepted.

How Much Effort Does It Take?

The researchers ran simulations using real data from X. They found:

You can trick the system on about 10.7% of lower-quality notes.
It takes less than 10 votes from a coordinated group to do it.
The cost? Roughly $30.50 per note (mostly just the cost of keeping the fake phone numbers active).

The Solution: The "Random Sample" Guard

The paper explains that the team at X has already fixed this. They added a new rule called "Population Sample Filtering."

The Analogy: Imagine a teacher grading a test. Instead of letting the whole class vote on the answer, the teacher picks a random handful of students from the back of the room to check the work.

If the "fake" group of attackers tries to vote, they might get picked.
But if they try to rig the vote, the system checks if the "random sample" agrees with the "whole class." If the random sample disagrees, the note doesn't get published.

This makes it much harder and more expensive for attackers to win, because they can't just target specific notes; they have to convince a random slice of the entire population, which is much harder to fake.

Summary

The paper shows that while the "Bridge" system is smart, it can be tricked by actors who learn to wear different masks. However, by understanding exactly how the trick works, the platform has added a new layer of defense (random sampling) to keep the bridge strong and the truth visible.

Technical Summary: Coordinated Manipulation in Crowdsourced Fact-Checking

Problem Statement
Crowdsourced fact-checking systems, such as X's Community Notes, Meta's Community Notes, and similar implementations by TikTok and Google, aim to combat misinformation at scale without centralized editorial control. These systems rely on a "bridging" mechanism: a note becomes publicly visible only if it receives support from users with diverse perspectives, rather than a simple majority. The core algorithm driving this consensus is matrix factorization, which learns low-dimensional latent representations (embeddings) for both users and notes.

The central problem addressed in this paper is the vulnerability of this matrix factorization component to coordinated manipulation. While bridging algorithms are theoretically resistant to simple volume-based attacks (e.g., fake upvotes), the authors investigate whether adversaries can strategically fabricate "synthetic consensus" by manipulating the latent factor space. Specifically, the paper asks if a coordinated group can control user accounts to simulate diverse viewpoints, thereby tricking the algorithm into assigning a high "helpfulness" intercept to a low-quality note.

Methodology
The authors develop a rigorous adversarial framework based on the open-source code and delayed data (48-hour lag) released by X Community Notes. The attack is modeled in two phases:

Phase 1: Establishing Diverse Factor Positions (The Inverse Problem)
- Goal: An adversary controls $k$ accounts and must position them at diverse locations in the latent factor space to simulate opposing viewpoints.
- Mechanism: Since user embeddings are determined by their rating history, the adversary solves an inverse problem. By predicting the latent parameters (factor $f_n$ and intercept $i_n$ ) of existing notes based solely on their text, the adversary can calculate the optimal rating (0, 0.5, or 1) to cast on those notes to steer their controlled accounts toward specific target factor vectors ( $f^{target}$ ).
- Prediction Model: The authors train a shallow Multilayer Perceptron (MLP) using Voyage embeddings of note text to predict the latent parameters ( $f_n, i_n$ ) that the matrix factorization algorithm would assign to a note.
Phase 2: Coordinating Support for Target Note
- Goal: Once the controlled accounts occupy diverse positions in the latent space, they coordinate to rate a specific target note $n^*$ as "Helpful."
- Mechanism: Because the algorithm perceives these accounts as representing diverse perspectives (based on their Phase 1 history), their unanimous positive rating is interpreted as "bridging agreement," driving the target note's intercept ( $i_{n^*}$ ) above the visibility threshold ( $\tau$ ).

Key Contributions

Novel Adversarial Attack: The paper presents the first rigorous analysis demonstrating that coordinated groups can manipulate the matrix factorization component to fabricate diverse agreement without exploiting software vulnerabilities.
Two-Phase Strategy: A formalized attack strategy involving the strategic positioning of accounts in latent space followed by coordinated voting.
Counterintuitive Theoretical Insight: The analysis reveals that, under specific geometric conditions of existing ratings, rating a note as "Not Helpful" (0) can mathematically increase its helpfulness intercept, contrary to intuitive expectations.
Manipulation Resistance Score (MRS): A metric quantifying the minimum number of additional ratings required to push a note above the consensus threshold.
Cost Model: A quantitative model estimating the effort (financial and computational) required to execute the attack, identifying account maintenance as the dominant cost factor.

Empirical Results
Using historic production data from X Community Notes (up to Jan 2025), the authors conducted simulations with 100 adversarial accounts:

Feasibility of Positioning: The adversarial accounts successfully positioned themselves across the factor spectrum (spanning $[-0.4, 0.4]$ ), validating the feasibility of Phase 1.
Manipulation Success: The study found that up to 10.7% of notes with intercepts below the median (and at least 10 existing ratings) could be manipulated above the consensus threshold using fewer than 10 coordinated ratings.
Cost Analysis: The estimated cost to manipulate a single note is approximately $30.50, dominated by the cost of maintaining accounts (approx. $3/month) rather than the cost of casting votes or training models.
Counterintuitive Finding: Theoretical derivation confirmed that in certain scenarios, a "Not Helpful" rating from an extreme voter can steepen the slope of the best-fit line in the latent space, thereby raising the note's intercept.

Significance and Claims
The paper claims to demonstrate a fundamental vulnerability in the core matrix factorization logic of bridging-based fact-checking systems. The significance lies in the revelation that the very mechanism designed to ensure diverse agreement (latent factor modeling) can be gamed if adversaries can predict note parameters and strategically curate their rating histories.

The authors emphasize that this work was conducted as an open research collaboration between Stanford University and the X Community Notes team. The findings were used to develop and deploy mitigations within X's production system prior to publication. Specifically, the paper highlights the deployment of population sample filtering, which recomputes quality scores based on a random subset of ratings to increase the attack cost linearly with the population size.

The authors conclude that while the matrix factorization component is vulnerable, the system's robustness can be enhanced through additional components (like abuse detection and sampling) that increase the cost of manipulation without sacrificing the transparency and openness that enable public research and improvement. The work serves as a call for further development of practical defenses that balance security with the accessibility required for effective crowdsourced fact-checking.