Technical Summary: COMMUNITYFACT

Problem Statement

Misinformation verification increasingly occurs in public, fast-moving, and multilingual online environments where static benchmarks fail to capture the dynamic nature of claims, evidence, and narratives. Existing factuality benchmarks often rely on fixed snapshots, centralized outlets, and English-centric data, leading to three critical gaps:

1. Coverage: Underrepresentation of localized and social-media-specific narratives in favor of professional fact-checkers and encyclopedias.
2. Granularity: Post-level labels often collapse multiple claims of mixed veracity into a single verdict, obscuring specific failure modes.
3. Redistributability: Social media posts are ephemeral due to deletion, access restrictions, and copyright, making raw data difficult to share and reproduce.

Furthermore, evaluating Large Language Models (LLMs) on static data risks measuring memorization or contamination rather than genuine verification capabilities, especially as LLMs are increasingly deployed as public, ad-hoc verification systems with web access.

Methodology

The authors introduce COMMUNITYFACT, a dynamic, multilingual, multi-domain benchmark constructed from X's (formerly Twitter) Community Notes. The system leverages the continuous stream of real-world claims and community-vetted corrective context provided by Community Notes to build a refreshable benchmark.

Data Construction Pipeline

The construction process involves a parameterized pipeline designed to ensure shareability and text-verifiability without requiring raw social media posts:

1. Source Selection: The pipeline filters the public Community Notes archive (snapshot through March 27, 2026) for notes marked CURRENTLY_RATED_HELPFUL created in 2025.
2. Language and Domain Filtering: It selects the five most frequent non-media note languages (English, Spanish, French, Japanese, Portuguese) and classifies notes into Politics and Finance domains using GPT-5.5.
3. Claim Extraction: Using GPT-5.5 with a "medium" reasoning effort, the system extracts standalone, checkworthy factual claims from Tweet–Note pairs. The extraction prompt enforces a "Zero-Context Formula," ensuring claims contain all necessary entities, dates, and qualifiers to be verified without access to the original post or note. Claims dependent on media or platform metadata are excluded.
4. Labeling: Labels are derived directly from the helpful Community Note. A claim is labeled FALSE if the note refutes or corrects a factual assertion in the post, and TRUE if the note supports or clarifies it.
5. Self-Refinement: A secondary audit step checks for standalone completeness, label correctness, and non-duplication.
6. Temporal Split: To prevent data leakage, examples are split temporally within each language–domain group, holding out the most recent examples for testing.

Benchmark Statistics

The 2025 release contains 15,992 standalone claim-level examples:

• Languages: English, Spanish, French, Japanese, Portuguese.
• Domains: Politics, Finance.
• Distribution: 9,970 TRUE claims and 6,022 FALSE claims.
• Format: Each example includes the claim, timestamp, language, domain, label, and evidence URLs cited in the note.

Experimental Setup

The authors evaluated 10 LLMs across four inference settings:

1. Instruction-following (Closed-input): Models receive only the claim and timestamp (e.g., Aya-Expanse, Ministral, EuroLLM).
2. Reasoning-enabled: Models run with explicit "thinking" modes (e.g., Qwen3 variants).
3. Web-search (Unguided): Models independently retrieve timestamp-relevant evidence.
4. Evidence-guided Web-search: Models are provided with URLs cited by the helpful Community Note as prioritized leads, while retaining access to general web search.

Evaluation metrics focus on macro-F1 scores, reported both overall and per language–domain slice to avoid masking systematic variations.

Key Contributions

1. A Refreshable Benchmark: COMMUNITYFACT is the first benchmark designed to be regenerated automatically from new Community Notes snapshots, addressing the staleness of existing datasets.
2. Granular, Shareable Claims: By converting Tweet–Note pairs into standalone claims with note-grounded labels and evidence URLs, the benchmark enables fine-grained analysis without redistributing raw social media content.
3. Multilingual and Multi-Domain Analysis: The dataset covers five languages and two high-impact domains, allowing for the evaluation of cross-lingual and cross-domain generalization.
4. Source-Selection Analysis: The benchmark facilitates the study of how web-enabled LLMs select evidence sources compared to human raters.

Results

Inference-Time Capabilities

• Closed-Input Limitations: Closed-input verification remains difficult. Even the best instruction-following model (Aya-Expanse-32B) achieved a macro-F1 of 69.93, significantly lower than web-enabled models.
• Reasoning Without Evidence: Explicit "thinking" modes did not universally improve performance; in some cases (e.g., Qwen3 variants, Gemini-2.5-Flash), reasoning without external evidence degraded performance.
• Web Access: Web-enabled models showed the largest gains. Grok-4.3 achieved the highest overall score (83.80 macro-F1) with web access, outperforming all closed-input models across every slice.

Evidence-Guided Search and Source Alignment

• Impact of Guidance: Providing evidence URLs from Community Notes significantly improved performance for GPT-5-nano (+1.48 pp) and Grok-4.3 (+1.97 pp), but did not significantly help Gemini-2.5-Flash (which actually dropped slightly).
• Source Misalignment: Web-enabled LLMs exhibit systematic misalignment with the sources human raters find helpful.
  • Grok-4.3 and GPT-5-nano successfully realigned their retrieval toward human-vetted sources (e.g., government domains, authoritative news) when guided, leading to performance gains.
  • Gemini-2.5-Flash failed to shift its source ecosystem toward human-preferred domains, relying instead on generic search endpoints (e.g., google.com, youtube.com), which resulted in no performance gain.
• Retrieval Efficiency: The gains were driven by which sources were consulted, not merely the quantity. GPT-5-nano expanded its sparse retrieval to match human citation density, while Grok-4.3 pruned its over-broad retrieval to focus on high-quality sources.

Significance and Claims

The paper positions COMMUNITYFACT not just as an evaluation tool, but as a training signal for future systems. The authors argue that:

• Static benchmarks are insufficient for measuring the reliability of deployed, web-enabled verification systems.
• Web access is critical for factual verification, but the quality of source selection is the differentiator.
• Community Notes provide a continuous, multilingual stream of <claim, human-verified sources> pairs. This suggests a future direction for training claim-conditioned source suggesters—lightweight retrieval priors that propose human-aligned sources for novel claims, bridging the gap between LLM retrieval policies and human verification standards.

The authors conclude that while aggregate scores mask significant variation across language and domain slices, the benchmark reveals that aligning retrieval policies with human-vetted sources is a key mechanism for improving misinformation detection in the wild.