Technical Summary: COMMUNITYNOTES

Problem Statement

Fact-checking on major social media platforms (e.g., X, Meta, TikTok) is transitioning from expert-driven verification to community-based systems where users provide explanatory notes to clarify misleading posts. A critical bottleneck in this ecosystem is determining whether a specific explanation is helpful for understanding real-world claims and, more importantly, identifying why it is helpful.

Current community note systems face two primary challenges:

1. Annotation Latency and Failure: The process of accumulating sufficient ratings is slow, causing over 90% of notes to remain unpublished.
2. Lack of Clear Definitions: While platforms provide reason categories for judgments (e.g., "helpful" vs. "unhelpful"), these reasons lack clear, unified definitions. This opacity hinders contributors from writing effective notes and raters from assessing them consistently, ultimately limiting the scalability and explainability of crowd-sourced fact-checking.

Prior research has largely underexplored the task of predicting both the helpfulness of an explanation and the specific reasons underlying that judgment.

Methodology

1. Dataset Construction: COMMUNITYNOTES

The authors introduce COMMUNITYNOTES, a large-scale, multilingual dataset derived from X's Community Notes platform (January 2021 – December 2024).

• Scale: Contains 104,966 posts with user-provided notes and aggregated helpfulness/reason labels.
• Composition: 58,569 English posts and 46,397 posts in other languages (dominated by Japanese and Spanish).
• Labels:
  • Helpfulness: Binary classification (Helpful/Unhelpful).
  • Reason: Multi-label classification across 18 predefined categories (8 for helpful notes, 10 for unhelpful notes).
• Preprocessing: The authors merged data shards, applied the official note-ranking algorithm to compute aggregated labels, and filtered out entries requiring more ratings (NEED_MORE_RATINGS).

2. Task Definition

The paper defines a dual-prediction task:

1. Helpfulness Prediction: A binary classification task to determine if a note explains the misleading nature of a post.
2. Reason Prediction: A multi-label classification task to identify the specific reasons (from the 18 categories) why a note is rated as it is.

3. Framework for Reason Definition Optimization

To address the subjectivity and lack of clarity in reason definitions, the authors propose a framework that automatically generates and optimizes these definitions using Automatic Prompt Optimization (APO).

• Seed Generation: Initial "seed" definitions for each reason category are generated by prompting GPT-4o with randomly sampled instances.
• Optimization: The PROMPTAGENT framework (based on Monte Carlo Tree Search) iteratively refines these seed definitions. It uses a feedback loop where a refiner agent improves prompts based on feedback from a feedback agent, optimizing for the reward signal derived from model performance.
• Integration (Fusion): The optimized definitions are embedded (using LLM2Vec) and fused with note representations via a Multi-Head Attention (MHA) mechanism. The note embedding acts as the query, while the reason definition embeddings act as keys and values. This allows the predictor to attend more strongly to relevant reason definitions during joint training.

4. Experimental Setup

• Models: Evaluated Small Language Models (SLMs) including BERT, ModernBERT, RoBERTa, and DeBERTa (base and large variants), as well as Large Language Models (LLMs) like Llama3.1-8B and Mistral-7B.
• Baselines: Standard fine-tuning and zero-shot prompting were compared against the proposed definition-fusion approach.
• Generalization: The utility of the predictors was tested on two external tasks:
  1. Evidence Sufficiency: Predicting if evidence is sufficient for a claim (mapping "sufficient" to "helpful").
  2. Automated Fact-Checking: Using predicted helpfulness scores as auxiliary signals for verifying climate change claims on the CLIMATE-FEVER dataset.

Key Results

Benchmark Performance

• Helpfulness Prediction: Models perform well, with most achieving F1 scores above 0.88. Larger models (LLMs) generally outperform SLMs (e.g., Mistral-7B achieved 0.920 F1).
• Reason Prediction: This task is significantly more challenging, with F1 scores consistently below 0.70. SLMs (specifically DeBERTa-large) generally outperformed LLMs on this specific subtask.

Impact of Reason Definitions

• Zero-Shot Improvement: Incorporating seed definitions generated by GPT-4o improved LLM performance by approximately 15% in F1 for both tasks. Further optimization via PROMPTAGENT yielded additional gains.
• Supervised Fusion: Integrating optimized reason definitions into SLMs via the MHA mechanism consistently improved performance across all models. For instance, DeBERTa-large-MHA achieved the highest Reason F1 score of 0.677, outperforming its vanilla counterpart.

Generalization

• Evidence Sufficiency: Helpfulness predictors showed transferable signals to the evidence sufficiency task. The top predicted reason for "unhelpful" notes (notHelpfulMissingKeyPoints) closely correlated with the "Not Enough Information" (NEI) label.
• Fact-Checking: Incorporating helpfulness information as an auxiliary signal improved automated fact-checking accuracy on CLIMATE-FEVER, with ModernBERT-base showing a statistically significant improvement (from 0.521 to 0.535 F1).

Significance and Claims

The paper claims that COMMUNITYNOTES and the proposed framework offer three primary contributions:

1. Resource Creation: It provides the first large-scale, multilingual dataset specifically designed to study the helpfulness and reasoning behind community fact-checking explanations.
2. Methodological Advancement: It demonstrates that automatically generating and optimizing reason definitions via prompt optimization significantly enhances model performance. This approach addresses the "opacity" of current community note systems by making evaluation criteria explicit and machine-interpretable.
3. Practical Utility: The study shows that predicting explanation helpfulness is not an isolated task; it provides valuable signals for related fact-checking challenges, such as determining evidence sufficiency and improving automated fact-checking systems.

The authors conclude that making evaluation criteria for explanation helpfulness more explicit and structured is crucial for the efficiency and scalability of community-based fact-checking. They position their dataset and framework as a foundation for building better predictors that can assist in assigning prior labels to real-time notes, thereby streamlining the verification process.

Limitations and Future Work

The authors acknowledge several limitations:

• Subjectivity: Community annotations inherently contain noise and cultural bias.
• Language Scope: The definition optimization pipeline was primarily evaluated on English; cross-lingual transferability was not systematically validated.
• Evaluation Metrics: Improvements were measured via classification metrics; human-centered evaluation of interpretability and practical utility is needed.
• Domain Specificity: The data is specific to X's ecosystem, and generalization to other platforms (Meta, TikTok) requires further investigation.

Future work aims to extend the framework to multilingual and multimodal settings and integrate it into end-to-end fact-checking systems.