🤖 machine learning

RiskNet: A large-scale dataset of AI risk incidents from news with alignment and multi-dimensional annotations

RiskNet is a large-scale, multilingual dataset of AI risk incidents derived from news sources that employs a structured pipeline for identification, alignment, and multi-dimensional annotation to support empirical research in AI safety, governance, and risk analysis.

Original authors: Leihan Zhang, Wecheng Ye, Xianlong Ma, Haochuan Liu, Yang Li, Qianyu Zhang, Jinliang Chen, Qiang Yan

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Leihan Zhang, Wecheng Ye, Xianlong Ma, Haochuan Liu, Yang Li, Qianyu Zhang, Jinliang Chen, Qiang Yan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the world of Artificial Intelligence (AI) as a massive, bustling city. Every day, new AI tools are built and put to work in hospitals, schools, banks, and on our phones. But just like in any big city, things sometimes go wrong. Cars crash, power grids fail, and people get hurt. In the AI city, these "accidents" are called AI risk incidents—things like a chatbot spreading lies, a facial recognition system misidentifying someone, or a self-driving car making a dangerous mistake.

Until now, trying to study these accidents was like trying to map a city's traffic jams by asking people to shout out what they saw from their windows. The information was scattered, messy, in different languages, and often just rumors or opinions rather than facts about specific crashes.

Enter "RiskNet."

Think of RiskNet as a giant, high-tech traffic control center built by researchers from Beijing University of Posts and Telecommunications. It's a massive digital library designed to catch, organize, and understand every AI accident reported in the news.

Here is how it works, broken down into simple steps:

1. The Great Filter (Finding the Accidents)

The researchers started with a mountain of news—hundreds of millions of articles from around the world, in many different languages.

The Problem: Most of these articles are just people talking about AI risks (like "AI might be dangerous someday") or general tech news. They aren't about a specific crash that actually happened.
The Solution: They used a smart "sieve" (powered by AI itself) to sift through the noise. It asked two questions:
1. "Is this about AI?"
2. "Did a specific accident actually happen here?"
- Analogy: Imagine a bouncer at a club. If you're just talking about how cool the music might be, you don't get in. But if you have a ticket for a specific show that already started, you're let through. RiskNet filters out the "talk" and keeps only the "action."

2. The Detective Work (Connecting the Dots)

Once they found the articles about real accidents, they faced a new problem: One accident often has many stories.

The Scenario: Imagine a robot vacuum catches fire in a kitchen. One newspaper in China writes about it. A blog in the US writes about it. A TV station in France reports it. They are all talking about the same fire, but they look like three different stories.
The Solution: RiskNet acts like a super-detective. It reads all these different reports and says, "Wait a minute, these three articles are describing the exact same event!" It then glues them together into a single "Incident Cluster."
Analogy: Think of it like a puzzle. You have 50 different puzzle pieces from different boxes, all showing parts of the same picture. RiskNet sorts them so you can see the whole picture of the accident, rather than just a confusing pile of fragments.

3. The Labeling System (Organizing the Files)

Now that they have a clean list of unique accidents, they need to organize them so researchers can find patterns.

They created a filing cabinet with specific tags for every accident:
- Who was hurt? (The victim)
- What went wrong? (The cause)
- How bad was it? (The severity)
- Where did it happen? (The location)
- What kind of AI was involved? (The tool)
Analogy: It's like a hospital's emergency room log. Instead of just writing "Patient came in," they write "Patient: John Doe; Injury: Broken leg; Cause: Bicycle accident; Severity: Moderate." This makes it easy to ask questions like, "How many bicycle accidents happened last month?"

What Did They Find?

The result is RiskNet, a dataset containing:

Hundreds of millions of news articles scanned.
Hundreds of thousands of AI-related risk reports.
Over 54,000 unique, verified accident clusters (where multiple news stories were merged into one event).

They discovered that most accidents get very little attention (like a small fender bender), but a few huge, famous accidents get thousands of news stories (like a major city-wide blackout). The data also shows that "lack of capability" (the AI just wasn't good enough) and "cyberattacks" are the most common types of problems.

Why Does This Matter?

The researchers say this dataset is a foundation stone for the future.

For Policymakers: It helps them see the real problems, not just the scary headlines, so they can write better rules.
For Scientists: It gives them a standardized way to test if new AI systems are safer than old ones.
For Everyone: It bridges the gap between "high-level principles" (like "AI should be safe") and the "messy reality" of what actually goes wrong.

Important Note: The paper emphasizes that this is a research tool, not a court of law. It organizes what the news says happened. It doesn't prove legal guilt or assign blame in a courtroom; it just helps us understand the landscape of AI risks so we can study them better.

In short, RiskNet turns a chaotic storm of news reports into a clear, organized map of where AI is stumbling, helping us figure out how to help it walk straight.

Technical Summary: RiskNet

Problem Statement

As artificial intelligence (AI) systems are increasingly deployed in high-stakes domains, reports of AI-related harms have grown in frequency and diversity. While high-level governance frameworks (e.g., the EU AI Act, U.S. AI RMF) articulate principles for responsible AI, there is a critical lack of large-scale, empirical resources for tracking and analyzing real-world AI risk incidents. Existing incident collections are often manually curated, small in scale, and insufficient for continuous, data-driven monitoring or downstream computational analysis. Furthermore, evidence is scattered across multilingual news reports with inconsistent descriptions, and existing tools often struggle with non-English or low-resource languages. There is a specific need for a resource that combines broad coverage, multilingual collection, incident-level alignment (aggregating multiple reports of the same event), and reusable benchmark annotations.

Methodology

RiskNet is constructed from large-scale, heterogeneous news sources through a structured pipeline comprising three main stages:

1. Data Sources

The dataset integrates content from multiple source types to ensure temporal coverage and linguistic diversity:

Public Incident Datasets: AIID (~~5,000 reports) and AIAAIC (~~4,500 reports).
Open News Corpora: CommonCrawl News (2022–2025, 674M articles), OpenNewsArchive (2023, 8.8M Chinese articles), and Hot-list Rankings (2021–2025, 340K trending topics).
Web Crawls: News websites (2024, ~800 articles).
Licensed Commercial Data: China News (2023–2025, 2.98M AI-related negative articles).

2. AI Risk Incident Identification

This stage filters raw news to identify event-level risk reports using a two-step process:

AI Relevance Filtering: A keyword-based filter (using a lexicon of 100+ terms) reduces the search space.
AI Risk Incident Determination: A Large Language Model (LLM) performs joint risk identification and structured attribute extraction. It distinguishes between event-level reports (containing grounded actors, systems, spatiotemporal context, and outcomes) and discussion-level reports (policy interpretation, commentary, or general debate). Only event-level reports are retained for alignment.
Extraction Schema: The LLM extracts attributes across three dimensions: AI System & Application, Event Context (time, location, actors, cause, outcome), and Risk & Impact (risk type, severity, harm type).

3. Incident Alignment

To aggregate reports referring to the same real-world incident, RiskNet employs a three-stage workflow:

Dual-View Candidate Recall: For each query report, the system retrieves candidates using two views: (1) Full-text recall (using BGE-M3 embeddings of the full article) and (2) Event-element recall (using embeddings of structured attributes like actors and causes). The top $K$ results from both views are merged.
Pairwise Same-Incident Prediction: A simplified DeepWide model predicts whether two reports describe the same incident. The "Wide" component uses explicit matching features (categorical agreement and text similarity), while the "Deep" component models higher-order semantic interactions via a Multi-Layer Perceptron (MLP).
Conservative Graph-Based Clustering: Positive pairwise predictions form edges in a graph. A complete-link constraint is applied: a report is added to a cluster only if it is positively linked to every existing member. This prevents error propagation and the formation of unrealistically large clusters.

4. Multi-dimensional Incident Classification

Aligned incidents are annotated using a unified taxonomy integrating the EU AI Act and MIT Media Lab taxonomies:

Risk Level: Unacceptable, High, Limited, Minimal (EU AI Act).
Domain Tags: 7 primary domains and 25 subdomains (MIT taxonomy).
Causal Tags: A triplet of Entity, Intent, and Timing.
Annotation Strategy: A hybrid approach where an LLM generates preliminary labels based on aggregated incident evidence (5 representative reports), followed by expert review and refinement.

Key Contributions

RiskNet Dataset: A large-scale dataset containing 777,100 AI risk-related reports, of which 264,776 are event-level reports. These are aligned into 54,386 incident clusters.
Multilingual Scope: The dataset covers mixed languages (primarily English and Chinese) and diverse source types, addressing the language gap in AI safety evaluation.
Structured Pipeline: A reproducible pipeline for AI risk news identification, event-level screening, incident alignment, and multi-dimensional classification.
Benchmark Subsets: The release includes benchmark datasets for event classification, incident alignment, and multi-dimensional incident classification, facilitating downstream research.
Platform: An online platform for browsing and exploring the data.

Results and Validation

Identification Performance: The AI relevance filtering achieved a recall of 0.9557. The LLM-based risk identification achieved a precision of 0.9550 and recall of 0.9717 on a manually annotated sample.
Alignment Performance: The incident alignment workflow achieved a Hungarian Macro-F1 of 0.895 on the test set. The pairwise classification achieved a positive F1 of 0.626.
Classification Performance: Fine-tuned LLMs (e.g., Qwen3-14B-sft) consistently outperformed prompt-based approaches. For single-label tasks (Risk Level, Intent), fine-tuned models achieved accuracies ranging from 0.79 to 0.93. For multi-label domain classification, fine-tuned models achieved Micro-F1 scores of 0.58–0.66.
Data Distribution: The incident cluster sizes follow a heavy-tailed distribution (scaling exponent $\alpha \approx 2.53$ ), where 29.8% of incidents are singletons, and the majority form small clusters (2–20 reports), reflecting the heterogeneous visibility of AI risk events.

Significance

RiskNet aims to bridge the gap between high-level governance principles and the documented realities of AI risk incidents. By providing a structured, reusable, and large-scale empirical resource, it supports:

Downstream research in AI safety, governance, risk analysis, and benchmarking.
Longitudinal and cross-source analyses of AI-related harms.
Cross-lingual and cross-domain studies of risk patterns.
The development of automated tools for incident tracking and risk assessment.

The authors emphasize that RiskNet is intended for research, monitoring, and aggregate analysis, rather than for automated high-stakes decision-making or formal legal adjudication. The dataset acknowledges that incident clusters are data-driven reconstructions and may reflect incomplete or evolving media coverage.