ClaimPT: A Portuguese Dataset of Annotated Claims in News Articles

1. Problem Statement

Fact-checking is a critical but labor-intensive process that struggles to keep pace with the rapid spread of misinformation. While automated fact-checking has emerged as a solution, progress is heavily skewed toward the English language and specific domains like social media or political debates.

• The Gap: There is a scarcity of high-quality, annotated datasets for Portuguese, particularly for journalistic content. Existing resources often focus on informal social media posts or parliamentary transcripts, which differ significantly from the structured, authoritative nature of news articles.
• The Challenge: Identifying "check-worthy" claims within news text is difficult because claims are often sparse, embedded within journalistic narratives, and require distinguishing between verifiable facts, opinions, and statements from inherently reliable sources (e.g., WHO) which do not require verification.

2. Methodology

Data Acquisition

• Source: The dataset was created in partnership with LUSA, the Portuguese News Agency.
• Corpus: 1,308 news articles published between January 2022 and December 2023.
• Topics: Covers nine domains (Politics, Society, Economy, International, Sports, Health, Culture, Science/Environment, Technology, Local).
• Selection Strategy: An iterative process prioritized articles with high claim density (Politics and International news) and filtered for sentences containing reporting verbs (e.g., "said," "declared") to target direct speech.

Annotation Scheme

The authors proposed a novel, multi-layer annotation scheme that extends previous frameworks (like NewsClaims and ClaimBuster) to capture the nuances of journalistic text:

1. Layer 1 (Metadata): Article-level topic and publication time.
2. Layer 2 (Core Classification): Distinguishes between Claims (verifiable factual statements of public interest) and Non-Claims (opinions, beliefs, predictions).
   • Constraint: Only statements attributed to non-official sources are annotated as claims; statements from inherently reliable institutional authorities are excluded.
3. Layer 3 (Fine-Grained Components):
   • Claim Span: The exact textual segment.
   • Claimer: The agent making the statement (Person, Organization, etc.).
   • Claim Object: The specific target of the assertion.
   • Time: Temporal context (explicit or implicit).
   • Stance: Affirmation or refutation.
   • Links: Attribute links (connecting components) and Identity links (resolving coreference, e.g., linking "he" to "Prime Minister").

Annotation Process

• Protocol: Two trained annotators independently labeled each article, followed by validation from an experienced curator.
• Quality Control: Disagreements were resolved through discussion, with a senior researcher making final decisions to ensure consistency.
• Inter-Annotator Agreement (IAA):
  • Claim/Non-Claim: Moderate agreement (F1 ~0.50–0.54), reflecting the inherent subjectivity of the task.
  • Attributes: High agreement on Span, Claimer, and Time (F1 > 0.75), but lower agreement on Claim Object (F1 ~0.44), indicating the difficulty in pinpointing the exact target of a claim.

3. Key Contributions

1. ClaimPT Dataset: The first large-scale, open-access dataset of European Portuguese news articles annotated for factual claims. It contains 1,308 articles and 6,875 annotations (463 claims, 4,393 non-claims).
2. Novel Annotation Scheme: A comprehensive framework that integrates metadata, entity attributes, and relational links (identity and attribute), specifically tailored for the structure of professional journalism.
3. Baseline Benchmarks: Establishment of initial performance benchmarks for Portuguese claim detection using state-of-the-art models.

4. Experimental Results (Baselines)

The authors evaluated the dataset using two types of models:

• Generative Models: Gemini 2.5 Flash and Flash-Lite (using few-shot prompting).
• Encoder Models: Fine-tuned BERTimbau (Portuguese BERT) using two strategies: Sentence-level segmentation and Chunking with overlap.

Performance Metrics (F1-Score):

• Generative Models: Performed poorly on the span classification task.
  • Gemini Flash-Lite: 22.52% (Micro Avg).
  • Gemini Flash: 36.50% (Micro Avg).
  • Analysis: These models struggled with boundary precision and often misinterpreted journalistic quotes as claims or failed to adhere to the "non-official source" constraint.
• Encoder Models (BERT): Significantly outperformed generative models.
  • BERT-Sentence Segmentation: Achieved the best results with 66.38% Micro Avg F1 (30.57% for Claims, 69.27% for Non-Claims).
  • BERT-Chunking: Achieved 59.43% Micro Avg F1.

Key Finding: The task is inherently difficult due to class imbalance (approx. 1 claim per 9.5 non-claims) and the requirement for precise span boundaries. Encoder models proved more robust for this specific extraction task than current generative LLMs.

5. Significance and Future Work

• Advancing Low-Resource NLP: ClaimPT fills a critical gap for Portuguese NLP, enabling research into fact-checking for a major global language that has been underrepresented.
• Real-World Applicability: By focusing on professional news articles rather than social media, the dataset supports the development of tools relevant to journalists and newsrooms.
• Future Directions:
  • Extending the annotation guidelines to other languages.
  • Improving baselines with newer architectures (e.g., ModernBERT).
  • Developing explainable models that leverage the rich attribute data (Claimer, Stance, Time) to better understand how claims are contextualized.

In conclusion, ClaimPT provides a foundational resource for automating the detection of misinformation in Portuguese news, offering both a high-quality dataset and a rigorous evaluation framework to guide future research in low-resource fact-checking.