💬 NLP

Misinformation Span Detection in Videos via Audio Transcripts

This paper addresses the challenge of pinpointing specific misinformation within videos by introducing two new datasets of annotated audio transcripts and demonstrating that state-of-the-art language models can effectively detect these misinformation spans with an F1 score of 0.68.

Original authors: Breno Matos, Rennan C. Lima, Savvas Zannettou, Fabricio Benevenuto, Rodrygo L. T. Santos

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Breno Matos, Rennan C. Lima, Savvas Zannettou, Fabricio Benevenuto, Rodrygo L. T. Santos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a very long, 55-minute documentary. Somewhere in the middle of it, the narrator tells a lie. Maybe they say, "The moon is made of cheese," or "Drinking bleach cures the flu."

The Problem:
Until now, if a computer wanted to check if that video was fake, it would have to watch the entire thing and give it a simple grade: "Pass" (it's true) or "Fail" (it's fake).

But that's like a teacher grading a whole essay just by saying "This is bad," without telling the student which sentence was the lie. If the video is an hour long, a human fact-checker has to sit there, watch the whole thing, and manually find the exact second the lie was spoken. That takes forever, and with millions of videos uploaded every day, it's impossible to keep up.

The Solution (The "Spot the Lie" Game):
This paper introduces a new way to play the game. Instead of just saying "This video is fake," the researchers built a system that acts like a highlight reel creator. It watches the video (specifically the audio) and says:

"Hey, the video is mostly fine, but between minute 12:05 and 12:20, the speaker is telling a lie."

They call this "Misinformation Span Detection." It's like finding the specific needle in the haystack, rather than just saying the whole haystack is made of needles.

How They Did It (The Recipe)

To teach the computer how to do this, the researchers had to create a special "training manual" because no such manual existed before.

Gathering the Evidence: They collected videos from two main sources:
- BOL4Y: A collection of videos featuring a former Brazilian president, Jair Bolsonaro, where fact-checkers had already identified thousands of false claims.
- EI22: Videos from the 2022 Brazilian election where voters claimed there was fraud.
Turning Speech into Text: They used a super-smart AI (called Whisper) to listen to the audio and write it down as text, breaking it into small chunks (like 30-second sentences).
The Matching Game: They took the list of known lies (e.g., "He built three power plants abroad") and compared them to the text chunks from the videos.
- Analogy: Imagine you have a list of stolen items (the lies) and a warehouse full of boxes (the video segments). The computer opens the boxes, reads the labels, and matches them to the stolen items list.
Human Verification: Since computers aren't perfect, human experts double-checked the matches to make sure the computer didn't get confused by similar-sounding words.

The Results: Did It Work?

They taught two different types of AI "brains" (language models) to spot these lies.

The Score: The AI managed to find the lies with a score of 0.68 (on a scale where 1.0 is perfect).
What that means: It's not perfect yet. It's like a detective who solves 68 out of 100 cases correctly. Sometimes it misses a lie, and sometimes it thinks a true statement is a lie. But it's a huge step forward because no one had tried to do this specific task before.

They also tested if the AI could learn from one type of video (Bolsonaro's speeches) and apply that knowledge to a totally different type of video (voters talking about election fraud). It worked surprisingly well, proving the system is flexible.

Why This Matters (The "So What?")

Think of social media platforms (like YouTube or TikTok) as massive libraries.

Before: If a librarian (the platform) found a book with a lie in it, they might have to ban the entire book, even if the lie was only on page 50. Or, they might miss it entirely because they don't have time to read every page.
After: With this new tool, the librarian can put a sticker on page 50 that says, "Warning: This paragraph is false," while letting the rest of the book stay open for people to read.

This helps:

Fact-checkers: They don't have to watch hours of video; the AI points them straight to the lie.
Viewers: They can see a warning label pop up exactly when the lie is spoken, giving them context immediately.
Regulators: New laws (like the Digital Services Act in Europe) require platforms to be more responsible. This tool gives them a way to do that without deleting everything.

The Bottom Line

The researchers didn't just build a detector; they built the first-ever map for finding lies inside long videos. They released their data and code to the world so other scientists can try to make the map even better. It's a bit like giving everyone a treasure map to find the "fake news" buried in the ocean of online video, rather than just telling them the ocean is dangerous.

1. Problem Definition

The paper addresses the challenge of Misinformation Span Detection in video content. While previous research has focused on binary classification (determining if an entire video contains misinformation), this approach lacks interpretability and utility for fact-checkers. It does not identify when or where within a video the false claims occur.

The authors define the task as identifying specific spans (segments) within a video transcript that contain false claims. This is crucial because a long video (e.g., a 55-minute interview) may be mostly factual but contain brief, critical moments of misinformation. Automating this detection can assist fact-checkers in pinpointing exact timestamps for verification and help social media platforms apply context labels or warnings at the precise moment a false claim is made.

2. Methodology

The authors propose a three-step methodology to construct datasets and train classifiers for this task:

A. Data Collection and Dataset Construction

The team created two novel datasets based on fact-checking work by Aos Fatos, a leading Brazilian fact-checking agency:

BOL4Y Dataset: Contains 538 videos featuring former Brazilian President Jair Bolsonaro. It includes 2,355 annotated false claims spanning his 4-year term.
EI22 Dataset: Contains 77 videos regarding electoral fraud claims made by voters during the 2022 Brazilian presidential election, with 78 annotated false claims.

Data Processing Pipeline:

Transcription: Audio from videos was converted to text using OpenAI's Whisper model. For some sources, transcripts were obtained directly from Aos Fatos' service (Escriba).
Segmentation: Transcripts were segmented into windows (Whisper segments are typically up to 30 seconds).
Embedding Generation: Segments and the original false claims were converted into dense vector embeddings using BERTimbau (a BERT model pre-trained on Brazilian Portuguese).
Segment Matching: The system calculated cosine similarity between claim embeddings and segment embeddings. Segments with high similarity (threshold $\ge$ 0.7) were flagged as candidates.
Manual Annotation: Human annotators verified candidate segments, flagging the exact span containing the claim. They also handled cases where claims spanned multiple segments by concatenating them.
Negative Sampling: All unannotated segments were treated as negative samples (non-misinformation).

B. Classification Models

The authors trained and evaluated two state-of-the-art language models pre-trained on Brazilian Portuguese:

BERTimbau: An encoder-only model.
PTT5: An encoder-decoder model based on the T5 architecture.

Both models were fine-tuned with a classification head (softmax) to predict whether a specific transcript segment contains a false claim (Class 1) or not (Class 0).

C. Experimental Setup

Class Imbalance Handling: Due to the massive number of negative samples (e.g., 336k negative vs. 2.3k positive in BOL4Y), the authors employed undersampling strategies with various positive-to-negative ratios (1:1, 1:10, 1:25, 1:50, 1:75, 1:100).
Dataset Variations: They tested an "Original" version (claims extracted from raw transcripts) and an "Edited" version (claims as polished by journalists), to assess the impact of transcription noise.
Temporal Analysis: They conducted "Walk-Forward" and "Expand" experiments to test if models trained on past data could predict future misinformation, simulating real-world deployment.
Cross-Dataset Evaluation: Models trained on BOL4Y were tested on EI22 to evaluate generalizability across different speakers and topics.

3. Key Contributions

First Span Detection Approach: This is the first work to frame and solve misinformation detection as a span detection problem in videos, moving beyond binary video-level classification.
Novel Datasets: The release of BOL4Y and EI22, containing over 2,400 annotated false claims with precise video timestamps, transcripts, and original media. These are the first datasets of their kind.
Baseline Establishment: The authors established the first benchmarks for this task, demonstrating that automated detection is feasible.
Open Science: All datasets (text, audio, video), code, and HTML scraping data are publicly released via Zenodo, HuggingFace, and GitHub, adhering to FAIR principles.

4. Results

Classification Performance:
- The best-performing model on the Original Dataset was BERTimbau with a 1:75 undersampling ratio, achieving a Macro F1 score of 0.68.
- Interestingly, training on the full dataset (without undersampling) yielded poor performance (F1 ~0.56), suggesting that extreme class imbalance hinders learning and that undersampling is critical.
- The Edited Dataset (cleaner, journalist-polished claims) showed improved performance, with BERTimbau reaching an F1 of 0.81, highlighting the challenge of noisy automatic transcriptions.
Temporal Analysis:
- Models demonstrated the ability to generalize to future months, though performance fluctuated based on political events (e.g., drops in performance during the 2022 election period).
- PTT5 showed more consistent performance across temporal settings compared to BERTimbau.
Cross-Dataset Performance:
- Models trained on BOL4Y successfully generalized to the EI22 dataset, achieving a Macro F1 of 0.71 (with 1:10 undersampling). This proves the approach works across different speakers and topics.

5. Significance and Impact

Practical Utility: The ability to pinpoint misinformation spans allows for contextual interventions. Platforms can overlay warning labels or fact-checks exactly when a false claim is spoken, rather than flagging the entire video.
Regulatory Compliance: The work supports compliance with regulations like the Digital Services Act (DSA), which requires platforms to have robust mechanisms for content moderation and transparency.
Research Foundation: By releasing high-quality, annotated datasets, the authors enable the research community to explore multi-modal strategies, Large Language Models (LLMs), and vision-language models for this specific problem.
Efficiency for Fact-Checkers: The tool can significantly reduce the time fact-checkers spend reviewing long videos by directing them to the specific segments requiring verification.

In conclusion, the paper successfully demonstrates that misinformation span detection in video transcripts is a feasible task using modern language models, provided that appropriate data handling (undersampling) and high-quality transcriptions are utilized.