🤖 machine learning

CrediBench: Building Web-Scale Network Datasets for Information Integrity

This paper introduces CrediBench, a web-scale multi-modal dataset comprising eight months of graph, text, and temporal data from over 40 million domains, which significantly enhances the performance of credibility prediction models by capturing structural and dynamic signals often missed by existing claim-level approaches.

Original authors: Emma Kondrup, Sebastian Sabry, Hussein Abdallah, Zachary Yang, Jiaqi Xiong, Kellin Pelrine, James Zhou, Zhijin Guo, Michael M. Bronstein, Jean-François Godbout, Reihaneh Rabbany, Shenyang Huang

Published 2026-06-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Emma Kondrup, Sebastian Sabry, Hussein Abdallah, Zachary Yang, Jiaqi Xiong, Kellin Pelrine, James Zhou, Zhijin Guo, Michael M. Bronstein, Jean-François Godbout, Reihaneh Rabbany, Shenyang Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a massive, bustling city where every website is a building. Some buildings are honest libraries, some are shady back-alley shops, and some are outright traps. Right now, it's getting harder to tell them apart because bad actors are building convincing fake storefronts, and new "buildings" pop up every second.

This paper introduces CrediBench, a giant new tool designed to help us navigate this city and figure out which buildings are trustworthy.

Here is the breakdown of what they did, using simple analogies:

1. The Problem: The Old Maps Are Too Small

Previously, researchers trying to spot fake news or dangerous websites had two main problems:

They only looked at the "menu": Most tools just read the text on a single webpage (the menu) to decide if it's a good restaurant. They ignored who the owner was or who they were friends with.
The maps were tiny: Existing datasets were like a map of a single neighborhood. They didn't have enough data to see the whole city, especially how the connections between buildings change over time.

2. The Solution: A Giant, Living City Map

The authors built CrediBench, which is like a high-definition, time-lapse map of the entire internet for eight months.

The Scale: It's massive. It tracks over 40 million "buildings" (domains) and 1 billion "roads" (hyperlinks) connecting them. To put that in perspective, it's orders of magnitude larger than any previous map of its kind.
The Three Layers: Instead of just looking at the text, CrediBench looks at three things at once:
1. The Text: What is written on the pages? (The menu).
2. The Structure: Who links to whom? (The neighborhood gossip and alliances).
3. Time: How do these connections change? (Did a reputable library suddenly start linking to a shady shop last week?).

3. The "Credibility Score"

The researchers taught computers to act like expert inspectors. They created two ways to test the system:

The "Trust Meter" (Regression): Instead of just saying "Good" or "Bad," the system gives a score from 0 to 1, indicating how credible a site is.
The "Pass/Fail" Test (Classification): A simple "Yes, this is safe" or "No, this is dangerous."

To train these inspectors, they didn't just guess. They gathered a massive list of 662,000 websites that had already been labeled by experts, crowds, and security firms as either "credible" or "unreliable" (covering things like misinformation, malware, and phishing). This is the largest list of its kind ever assembled.

4. The Results: Teamwork Wins

The team ran experiments to see which "sense" was most important: reading the text, looking at the links, or watching the timeline.

The Finding: You can't rely on just one sense. A model that only reads text is okay, but a model that only looks at links is also okay.
The Winner: The Super-Inspector that combines all three (Text + Links + Time) was the clear champion.
- On the "Trust Meter" task, it reduced errors by a huge margin (dropping from 16% error down to 10%).
- On the "Pass/Fail" test, it jumped from being right 56% of the time to being right 85% of the time.

5. Why This Matters

The paper argues that misinformation spreads like a virus through the connections between websites, not just through the words on a page. By looking at the whole city map and how it moves over time, we can spot the "bad neighborhoods" much faster.

In short: The authors built the biggest, most detailed map of the internet's trust network ever created. They proved that to spot a liar, you need to read what they say, see who they hang out with, and watch how their relationships change over time. They made this map and the training tools available for anyone to use, so future researchers can build better "city inspectors" to keep the internet safe.

Technical Summary: CrediBench: Web-Scale Network Data for Credibility Prediction

Problem Statement

The automatic assessment of online source credibility is critical for navigating modern information ecosystems, particularly given the rapid spread of misinformation and the emergence of generative AI, which can produce indistinguishable and persuasive content. Existing approaches to credibility prediction suffer from two primary limitations:

Data Scarcity and Cost: Most methods rely on scarce, costly human annotations or expert-driven evaluations that do not scale to the volume of modern information flows.
Modality Narrowness: Current research predominantly focuses on individual claims, posts, or documents, analyzing only text-based (endogenous) signals. This ignores the structural and temporal signals inherent in the web's topology, such as interlinked domains and their evolution over time.

Furthermore, existing datasets for credibility prediction are limited in scale (often containing only tens of thousands of domains) and frequently lack the integration of three central modalities: internet topology (graph structure), temporality (evolution over time), and webpage text content. This gap hinders the development of scalable, generalizable mitigation strategies for misinformation.

Methodology

The authors present CrediBench, a web-scale dataset and framework designed to address these gaps by modeling credibility prediction as a Text-Attributed Temporal Graph (TAG) learning problem.

1. Data Construction Pipeline

CrediBench is constructed from eight months of Common Crawl data. The pipeline involves:

Graph Construction: Nodes represent web domains, and edges represent hyperlinks between them. The dataset includes over 40 million nodes and over 1 billion edges per monthly snapshot.
Text Extraction: Raw webpage content (plain body text) is extracted for each domain from Common Crawl WET files using a distributed Spark cluster. This text is processed into embeddings.
Temporal Snapshots: The data is organized into monthly snapshots (e.g., October–December 2024), capturing the dynamic evolution of the web graph and content.
Filtering: Domains with a degree below a threshold (defaulting to 3) are filtered to maintain a single connected component of active domains, reducing computational load and focusing on relevant entities.

2. Label Sets

To enable supervised learning, the authors curated two distinct label sets:

Regression Labels (Domain Quality Ratings - DQR): A dataset of 11,520 domains labeled with expert-driven credibility scores. The ground truth is the First Principal Component (PC1) score derived from a PCA of six expert sources, ranging from 0 to 1.
Binary Classification Labels (DomainRel): A novel dataset of 662,575 domains aggregated from eight sources (including fact-checkers, crowd-sourced lists like Wikipedia, and malware/phishing databases). Domains are labeled as either credible (1) or unreliable (0), covering four categories: misinformation, crowd-sourced, phishing, and malware.

3. Task Formulation

The paper formulates credibility prediction as two node-level tasks on the TAG:

Regression: Predicting a continuous credibility score ( $y \in [0, 1]$ ).
Binary Classification: Predicting a boolean credibility status ( $y \in \{0, 1\}$ ).

4. Experimental Models

The authors evaluate various modalities and combinations:

Text-Only: Using Large Language Models (LLMs) like Gemma3, Qwen3, and OpenAI TE3L to embed text content, fed into MLP regressors/classifiers.
Graph-Only: Using Graph Neural Networks (GCN, GraphSAGE, GAT) to learn hyperlink structures.
Multi-Modal: Concatenating graph embeddings (from GAT) and text embeddings (from LLMs) to form joint representations.
Temporal: Aggregating embeddings across consecutive monthly snapshots to capture temporal evolution.

Key Contributions

Largest Domain Credibility Dataset: CrediBench is the largest dataset of its kind, featuring 8 months of web-scale graphs with rich text attributes. Each snapshot contains >40M nodes, >1B edges, and >100GB of text content.
Novel Multi-Modal Labeling: The introduction of DomainRel, a binary labeled set of 662k domains spanning four distinct credibility categories, significantly outstrips existing domain-level datasets in size and diversity.
Empirical Validation of Multi-Modality: The study demonstrates that combining graph structure, text content, and temporal evolution yields superior performance compared to single-modality approaches.
Open Resources: The dataset is hosted on Hugging Face, and a user-friendly PyPI package (credigraph) is provided for API access and reproducibility.

Experimental Results

Experiments were conducted on snapshots surrounding the 2024 U.S. federal elections (October–December 2024).

Regression Performance:
- The multi-modal model (GAT + Text Embeddings) achieved the best results, reducing the Mean Absolute Error (MAE) from 0.162 (LLM-URL baseline) to 0.107.
- Text-only models (using OpenAI TE3L) outperformed graph-only models, indicating that text content provides strong signals, but the combination of both modalities yields the lowest error.
Classification Performance:
- The multi-modal classifier improved accuracy from 56% (LightGBM baseline) to 85%.
- The best performing model was a multi-modal GAT with text embeddings (GAT ⊕ Gemma3/TE3L), achieving ~84-85% accuracy across snapshots.
Temporal Insights:
- Incorporating temporal evolution (aggregating data across months) further reduced MAE to 0.107 on the regression task, confirming that the dynamic nature of the web graph is a critical signal for credibility.
- Analysis of domain overlap showed that while ~49% of domains are retained across months, only ~19% of edges are common, highlighting the necessity of temporal graph learning.

Significance and Claims

The paper claims that CrediBench provides a "strong foundation for future research on credibility prediction" by addressing the critical lack of large-scale, multi-modal datasets. The authors assert that:

Scalability is Key: Existing methods fail to scale to the web level; CrediBench enables the training of models capable of handling billions of interactions.
Holistic Signals are Necessary: The empirical results support the hypothesis that credibility is not solely a function of text or structure, but a result of their interplay over time. Ignoring any of these modalities leads to suboptimal performance.
Reproducibility: By releasing the data and code, the authors aim to accelerate the development of principled, scalable mitigation strategies against misinformation, particularly in the context of evolving content generation methods like LLMs.

The work positions itself as a necessary step toward understanding the "structural and temporal credibility signals evident in the changing web topology," moving beyond the limitations of claim-level analysis to a domain-level, web-scale perspective.