💻 computer science

SciClaimSeekers at CheckThat! 2026: Retrieving Scientific Sources for Social Media Claims with LLM Reranking

The SciClaimSeekers system presented in this paper addresses the challenge of verifying social media scientific claims by employing a hybrid retrieval pipeline combining BM25 and multilingual E5 with Reciprocal Rank Fusion, followed by Qwen2.5-14B-Instruct reranking, which achieved a 64.39% MRR@5 on the CLEF-2026 CheckThat! Task 1 test set and demonstrated that such carefully constructed pipelines can rival fine-tuned approaches.

Original authors: Mohotarema Rashid, Nansu Baniya, Anirban Saha Anik, Xiaoying Song, Lingzi Hong

Published 2026-07-29

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Mohotarema Rashid, Nansu Baniya, Anirban Saha Anik, Xiaoying Song, Lingzi Hong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, chaotic library where anyone can shout out a "fact" from the top of a tower. Sometimes, these shouts are true, but often they are just rumors or misunderstandings of real science. The problem is that when people share these scientific ideas on social media, they rarely bring the original book or the expert's report with them. It's like someone saying, "I read that eating broccoli gives you super strength," but refusing to tell you which book they read it in. This makes it hard to check if the story is true or just a made-up myth. Scientists and fact-checkers need a way to find the original "source code" of these claims—the actual research papers hidden in the massive library of academic writing. This is the challenge of "source retrieval": taking a messy, informal social media post and finding the one specific, formal scientific paper it was trying to talk about.

The paper you are about to read, titled "SciClaimSeekers at CheckThat! 2026," describes a clever new system built to solve this exact puzzle. The researchers created a digital detective team called SciClaimSeekers. Think of this system as a three-step process to find the right book in a library of 10,000 scientific papers. First, it uses two different "search engines" at the same time: one that looks for exact word matches (like finding a book because the title has the word "broccoli" in it) and another that understands the meaning of the sentence (like finding a book because it talks about "super strength" even if the word "broccoli" isn't there). Second, it combines the lists from both search engines to create a single, shorter list of the most likely candidates. Finally, and most importantly, it uses a powerful Artificial Intelligence (AI) brain to read the social media post and the top candidates one by one, asking, "Is this the paper this post is talking about?" and giving it a score.

The team found that this "AI brain" step was the real magic. While the first two steps helped narrow things down, the AI reranking step boosted the system's success rate significantly. On a test set of social media posts, their system correctly identified the right scientific paper in the top five guesses about 64.39% of the time. This was a huge jump—about 13.6 percentage points better than using just the simple word-matching search engine. The results suggest that using these large, pre-trained AI models in a careful, step-by-step pipeline is a very strong way to verify scientific claims, even without needing to train a custom AI from scratch. The system is so effective that it ranked 11th out of 37 teams in a major international competition, proving that this mix of simple search and smart AI reading works well to connect social media noise with scientific truth.

Technical Summary: SciClaimSeekers at CheckThat! 2026

Problem Statement
Scientific claims frequently spread on social media faster than they can be verified, yet social media posts rarely link to their original scholarly sources. This creates a significant challenge for automated fact-checking systems. The difficulty is compounded by "confirmation bias," where users favor information aligning with preexisting beliefs, and the structural mismatch between informal, jargon-heavy social media discourse and formal scientific literature. Furthermore, user-generated content often paraphrases findings or references them loosely without standardized citations, making the identification of original scientific sources difficult. The CheckThat! 2026 Task 1 specifically addresses the retrieval of scientific sources for social media claims, requiring systems to map a social media post to the correct publication from a fixed pool of 10,000 scientific papers.

Methodology
The authors propose SciClaimSeekers, a three-stage "retrieve-then-rerank" pipeline designed to bridge the gap between noisy social media posts and formal scientific abstracts. The system operates as follows:

First-Stage Retrieval (Hybrid): Two complementary retrievers operate in parallel on an indexed corpus of 10,000 papers (represented by concatenated title and abstract):
- Sparse Retrieval: Uses BM25 (Okapi) to capture lexical signals, such as named entities and numerical values. The input is preprocessed to remove noise (URLs, mentions, punctuation) and normalized.
- Dense Retrieval: Uses the multilingual-e5-large-instruct model (based on XLM-RoBERTa). This model employs instruction prefixes to differentiate queries from documents, treating the social media post as a query to retrieve the corresponding scientific paper. It utilizes mean pooling over final hidden states and L2 normalization for cosine similarity matching via a FAISS index.
- Fusion: Both retrievers return their top-100 candidates. These lists are merged using Reciprocal Rank Fusion (RRF) with a smoothing constant $k=60$ , producing a unified top-50 candidate set.
Second-Stage Reranking (LLM): The fused top-50 candidates are reranked using Qwen2.5-14B-Instruct, an open-source Large Language Model.
- Approach: The model is used as a zero-shot pointwise cross-encoder. Unlike listwise approaches, each (social media post, candidate paper) pair is scored independently.
- Prompting: The system instructs the LLM to act as a scientific source identifier, outputting an integer relevance score between 0 and 10 based on the likelihood that the post refers to the specific paper.
- Output: Candidates are sorted by the LLM score (with ties broken by the Stage 2 RRF score), and the top 5 are returned as the final prediction.

Key Contributions

Hybrid Pipeline Performance: The authors present a hybrid retrieve-then-rerank pipeline that achieved 11th place out of 37 participating teams in the English test set of the CheckThat! 2026 Task 1. The system achieved an MRR@5 of 64.39% on the official test set.
LLM Dominance in Reranking: The study isolates the contribution of each stage, identifying the LLM cross-encoder as the primary driver of performance. The Qwen2.5-14B reranker provided approximately +10 MRR@5 points whether applied to BM25 candidates alone or the hybrid fused candidates. This accounts for roughly three-quarters of the total +13.67-point improvement over the BM25 baseline.
Generalization: The system demonstrated strong generalization, with a negligible 0.03 percentage-point gap between the development set score (64.36%) and the test set score (64.39%), suggesting that the tuning decisions transfer cleanly to held-out data.

Results and Analysis
The evaluation, based on Mean Reciprocal Rank (MRR@5), Hit@1, and Hit@5, reveals the following:

Baselines: BM25 alone achieved ~50% MRR@5. The dense E5 retriever improved this by ~2 points, and the hybrid RRF fusion added another ~1.5 points.
Reranking Comparison: A traditional cross-encoder (bge-reranker-v2-m3) showed modest gains on the development set but suffered from overfitting, performing worse than the hybrid baseline on the test set. In contrast, the generative LLM reranker (Qwen2.5-14B) delivered substantial, consistent gains.
Efficiency vs. Performance: While the LLM reranker is the computational bottleneck (taking ~11 hours for the test set on two A40 GPUs), the authors note that the pipeline is modular and runs end-to-end on a single node, making it practical for offline or privacy-sensitive deployments.

Significance and Claims
The paper claims that large pre-trained models, when integrated into a careful pipeline, can be competitive with fine-tuned approaches for scientific source retrieval. Specifically, the authors argue that pointwise scoring with a strong open-weight LLM serves as a reliable substitute for training a task-specific reranker. The work highlights that while hybrid retrieval (combining sparse and dense signals) provides a necessary foundation for recall, the significant leap in ranking quality is driven by the semantic understanding capabilities of the LLM in the reranking stage. The system is presented as an open-source, modular solution that effectively addresses the linguistic and structural mismatches inherent in linking social media claims to scientific evidence.

Technical Summary: SciClaimSeekers at CheckThat! 2026

More like this