💬 NLP

ClimateCheck 2026: Scientific Fact-Checking and Disinformation Narrative Classification of Climate-related Claims

ClimateCheck 2026 is a shared task that advances the automated verification of climate-related claims and the classification of disinformation narratives through expanded datasets, novel evaluation frameworks addressing annotation biases, and insights into the varying verifiability of different climate misinformation types.

Original authors: Raia Abu Ahmad, Max Upravitelev, Aida Usmanova, Veronika Solopova, Georg Rehm

Published 2026-03-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Raia Abu Ahmad, Max Upravitelev, Aida Usmanova, Veronika Solopova, Georg Rehm

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a librarian in a massive, chaotic library where millions of people are shouting claims about the weather, the ice caps, and the future of our planet. Some shouts are true, some are lies, and some are just guesses. Your job is to find the right books (scientific papers) to prove or disprove these shouts.

This paper, ClimateCheck 2026, is a report card on a competition where computer programs tried to do this librarian job. Here is the story of what happened, explained simply.

1. The Big Problem: The "Noise" vs. The "Signal"

For a long time, fact-checkers have been like detectives looking for clues in a messy room. They usually check general sources like Wikipedia or news websites. But for climate change, that's not enough. You need to go straight to the source: the scientific abstracts (the summaries of serious research papers).

The problem? Scientific papers are written in "robot language" (complex jargon), they are long, and they change over time. Meanwhile, climate misinformation on social media is loud, emotional, and often uses clever tricks to sound true.

2. The Competition: "ClimateCheck 2026"

The organizers (a team of researchers from Germany) set up a giant challenge called ClimateCheck 2026. They gave 20 teams of AI experts a massive pile of data—three times bigger than last year's challenge—and asked them to build robots that could:

Task 1: The Detective Work. Given a social media claim (e.g., "Glaciers aren't melting"), find the top 5 scientific papers that talk about it. Then, decide: Does the paper Support the claim, Refute (prove it wrong), or is there Not Enough Info?
Task 2: The Mind Reader. Instead of just checking the fact, the robot had to guess the story behind the lie. Is the person spreading misinformation because they think "Climate change is a hoax," or because they think "The solutions are too expensive," or "Science is unreliable"?

3. The New Twist: Finding the "Story"

In previous years, the focus was just on "True or False." This year, they added a new layer: Narrative Classification.

Think of it like this:

Old Way: A robot sees a claim and says, "False."
New Way: A robot sees a claim and says, "False. And I know why you are saying this. You are using the 'It's Natural Cycles' story, which is a common script used by deniers."

This is a huge deal because it helps us understand how misinformation spreads, not just that it exists.

4. The Results: Who Won?

Eight teams submitted their best robots. Here's what they found:

The "Smart Search" Won: The best robots didn't just use one big brain (a giant AI model). They used a multi-stage pipeline. Imagine a hiring process:
1. The Screener: Quickly scans thousands of papers to find a few hundred that might be relevant (like a resume screener).
2. The Interviewer: Reads those few hundred closely to pick the top 5.
3. The Judge: Reads the final 5 and the claim together to make the final verdict.
- Lesson: Being smart about how you search is more important than just having a huge computer.
The "Story" was Harder: While the robots got pretty good at finding the right papers, they struggled to classify the specific "story" (narrative) of the lie. It's like being able to find the right book, but having trouble understanding the author's hidden motive.

5. The Big Surprise: Some Lies are Harder to Prove

The researchers discovered something fascinating: Not all lies are created equal.

Easy Lies: "The sea level isn't rising." (Easy to prove wrong with a simple graph).
Hard Lies: "Climate science is unreliable." (This is a "meta-lie").
- The Analogy: If someone says, "The librarian is lying about the books," and you try to prove them wrong by showing them a book, they just say, "See? The book is part of the lie!"
- The robots found it very hard to refute claims that attack the credibility of science itself. You can't use a scientific paper to prove that science is trustworthy if the claim is that the paper is fake. This is a "structural trap" for AI.

6. The "Hidden Score" Problem

The paper also points out a flaw in how we usually grade these robots.

The Official Score: If a robot finds a paper that is "correct" but wasn't in the official answer key, the robot gets punished.
The New Score (Ev2R): The researchers invented a new way to grade. They used a "proxy" (a smart AI judge) to check if the robot found good evidence, even if that evidence wasn't in the original answer key.
The Result: One team that looked like a loser on the official scoreboard actually found better evidence than the winners, but the old scoring system didn't see it. This teaches us that we need better ways to grade AI so we don't punish it for being too thorough.

7. The Takeaway

ClimateCheck 2026 taught us three main things:

Structure matters: The best AI systems use a step-by-step process (search, refine, judge) rather than just guessing.
Context is key: To fight misinformation, we need to understand the story behind the lie, not just the facts.
Some battles are harder: AI is great at checking facts, but it struggles when the lie is about the system of facts itself.

In short, we are building better "librarians" for the climate age, but we still have work to do to teach them how to handle the trickiest, most confusing stories.

1. Problem Statement

The paper addresses the critical challenge of Automated Fact-Checking (AFC) regarding climate change claims. While AFC has advanced, most systems rely on general web sources (e.g., Wikipedia) rather than authoritative scientific literature. Verifying claims against scholarly abstracts is difficult due to:

Specialized Terminology: Domain-specific jargon in scientific papers.
Complex Reasoning: The need to connect claims to evidence through citations and logical inference.
Disinformation Narratives: Climate misinformation often operates through recurring rhetorical patterns (narratives) rather than isolated false claims, making simple claim-level verification insufficient.
Annotation Limitations: Incomplete ground truth annotations in existing datasets bias evaluation metrics.

2. Methodology and Dataset

The authors present ClimateCheck 2026, a shared task iteration that expands upon the 2025 edition.

Dataset Development

Scale: The training data was tripled, adding 523 unique English claims and 1,897 Claim-Abstract Pairs (CAPs). The final dataset contains 958 unique claims and 4,927 CAPs.
Annotation:
- Task 1 (Verification): Claims are linked to 1–5 abstracts and labeled as Supports, Refutes, or Not Enough Information (NEI). Inter-annotator agreement (IAA) is substantial ( $\kappa = 0.73$ ).
- Task 2 (Narrative Classification): The same claims are annotated with Disinformation Narratives based on the Coan et al. (2021) taxonomy. This includes 27 specific labels across 5 main groups (e.g., "Ice isn't melting," "Solutions won't work"). A multi-label scheme was used, with IAA measured via Krippendorff's $\alpha$ (ranging from 0.55 to 0.72 depending on the label).

Task Definitions

Task 1.1 (Retrieval): Retrieve the top 5 most relevant scientific abstracts for a given claim.
Task 1.2 (Verification): Classify the relationship between the claim and each retrieved abstract (Supports/Refutes/NEI).
Task 2 (Narrative Classification): Predict the disinformation narrative(s) present in a claim using a hierarchical taxonomy.

Evaluation Framework

Standard Metrics: Recall@K, Binary Preference (Bpref), and F1 scores.
Adapted Evaluation (Ev2R): To address incomplete annotations (where retrieved abstracts may be relevant but not in the gold set), the authors adapted the Ev2R framework. This uses:
- A reference-based component: Decomposing texts into atomic facts to measure alignment.
- A proxy-reference component: Using a fine-tuned DeBERTa model to predict veracity and confidence for unannotated pairs.
- This allows for a more robust assessment of retrieval quality beyond the limited gold standard.

3. Key Contributions

Unified Benchmark: The first dataset to simultaneously connect evidence retrieval, claim verification, and narrative classification over the same set of claims, enabling cross-task analysis.
Robust Evaluation Framework: An adaptation of the Ev2R framework to handle incomplete annotations in multi-evidence scientific fact-checking, revealing biases in standard metrics.
Cross-Task Empirical Analysis: A novel investigation into how disinformation narratives influence the difficulty of retrieval and verification, revealing structural limitations in current AFC approaches.

4. Results and System Performance

The competition ran on CodaBench with 20 registered participants and 8 leaderboard submissions.

Task 1: Retrieval and Verification

Top Performers: Systems like ClimateSense and Ant Bridge utilized multi-stage pipelines (BM25 $\to$ Dense Re-ranking with BGE/MiniLM $\to$ Cross-Encoder).
Verification: Zero-shot LLMs (e.g., GPT-4o-mini, gpt-oss-120b) often outperformed fine-tuned smaller models, particularly in reasoning tasks.
Key Finding: While retrieval performance was generally high, verification of "Refutes" labels remained the most difficult, with systems frequently confusing "Refutes" with "Supports" or "NEI."

Task 2: Narrative Classification

Top Performers: ahilbert and XplaiNLP achieved the highest Macro-F1 scores.
Strategy: The most effective approach involved hierarchical reasoning (identifying broad narrative groups before fine-grained sub-labels) and Retrieval-Augmented Generation (RAG) using the taxonomy.
Efficiency Trade-off: Reasoning-heavy approaches (Chain-of-Thought) improved accuracy but consumed ~25x more energy than zero-shot inference.

Ev2R Analysis (Unannotated Data)

The system ytsoneva ranked last on official metrics but achieved the highest Ev2R score. This system retrieved many unannotated abstracts that were actually high-quality evidence, demonstrating that standard metrics penalize systems for retrieving evidence outside the limited gold set.

5. Significance and Insights

The paper provides several critical insights for the future of scientific fact-checking:

Narrative-Dependent Verification Difficulty:
- Retrieval is narrative-agnostic: Relevant scientific evidence can be found regardless of the disinformation narrative type.
- Verification is narrative-dependent: Claims belonging to Group 4 ("Solutions won't work") and Group 5 ("Science is unreliable") are systematically harder to refute using scientific abstracts.
- Structural Limitation: Group 5 claims attack the credibility of science itself; therefore, using a scientific abstract to refute them is "self-referentially problematic." This suggests that standard evidence-grounded AFC has inherent limits for epistemological claims.
Future Directions:
- Narrative-Aware Pipelines: Systems should first classify the narrative type and then adapt their verification strategy (e.g., flagging epistemological claims for human review rather than automated verdicts).
- Beyond Gold Annotations: Evaluation frameworks must evolve to handle incomplete annotations, as exhaustive annotation of scientific evidence is rarely feasible.
- Hierarchical Reasoning: Success in narrative classification relies on coarse-to-fine reasoning strategies rather than just model size.

Conclusion

ClimateCheck 2026 demonstrates that while automated systems can effectively retrieve scientific evidence, the veracity classification of climate disinformation is heavily influenced by the underlying rhetorical narrative. The paper argues for a shift from purely evidence-grounded verification to narrative-aware verification, acknowledging that some forms of climate misinformation are structurally resistant to standard scientific refutation.