💬 NLP

The MediaSpin Dataset: Post-Publication News Headline Edits Annotated for Media Bias

This paper introduces the MediaSpin dataset, a comprehensive collection of 78,910 post-publication news headline pairs annotated with 13 types of media bias, designed to analyze how editorial edits shape public perception and enable bias prediction.

Original authors: Preetika Verma, Kokil Jaidka

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Preetika Verma, Kokil Jaidka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a live news broadcast. The anchor reads a headline, but then, just a few minutes later, they pause, look at their script, and say, "Actually, let me tweak that." They change a word here, add a phrase there, or delete a detail. To you, the viewer, it might seem like a tiny correction. But to the researchers behind this paper, that tiny tweak is a goldmine of information.

Here is the story of MediaSpin, explained simply.

The "Magic Edit" Button

Think of online news like a Google Doc that never stops being edited. In the old days, once a newspaper was printed, the story was frozen in ink. Today, news outlets can hit "save" and then hit "edit" again and again.

The researchers (Preetika Verma and Kokil Jaidka) realized that these edits are like a "behind-the-scenes" movie of how news is made. They built a massive library called MediaSpin that tracks 78,910 pairs of headlines: the "Original Draft" and the "Final Edited Version."

They didn't just look at what changed; they asked, why did it change? Did the editor make the story sound more exciting? Did they leave out a boring but important fact? Did they swap a neutral word for an emotional one?

The 13 "Flavors" of Bias

To understand these changes, the researchers created a menu of 13 types of "flavors" (or biases) that editors might accidentally or intentionally add to a story. Think of it like a chef tasting a soup and deciding to add too much salt, or leave out the garlic.

Here are a few examples of these flavors:

Sensationalism (The "Popcorn" Effect): Changing "A car crashed" to "A terrifying crash!" to make your heart race.
Mind Reading (The "Crystal Ball"): Changing "The politician voted no" to "The politician, who clearly didn't care, voted no." (The editor is guessing the politician's feelings).
Omission (The "Magic Eraser"): Removing the name of the source. Instead of "The Health Department says...", it just becomes "Millions could get sick."
Spin (The "Reframing"): Changing "Protesters blocked the road" to "Rioters blocked the road." The action is the same, but the feeling is totally different.

How They Did It (The Robot Chef)

Analyzing 78,000 headlines by hand would take a human team years. So, the researchers used a super-smart AI robot (a Large Language Model) to read every headline pair and tag the changes.

To make sure the robot wasn't hallucinating, they had human experts taste-test the soup. They checked about 500 examples, and the humans agreed with the robot about 85% of the time. It wasn't perfect (robots sometimes miss subtle context), but it was good enough to build a massive map of how news changes.

What They Discovered: The "Geography Game"

When they looked at the data, they found some fascinating patterns, like a game of "Hide and Seek" with countries:

The "Big Names" Get Bigger: When headlines were edited, countries like the USA, China, and Russia were often added to the headline. Editors seemed to want to make sure the big players were named.
The "Small Names" Get Smaller: Countries like Belgium, the Philippines, or Cuba were often removed or replaced with vague terms like "Europe" or "Asia."
- Analogy: Imagine a spotlight on a stage. The editors are moving the spotlight away from the smaller actors (small countries) and shining it intensely on the main stars (superpowers), making the smaller actors fade into the background.

The "Clickbait" Effect

The researchers also looked at what happens after the headline is edited. They tracked how people reacted on X (formerly Twitter).

They found a simple rule: The more biased the headline, the more people clicked, liked, and shared it.

If an editor added an emotional word (like "horrifying"), people shared it more.
If an editor removed a boring source citation, people shared it more.

It's like a carnival barker. The more dramatic the pitch, the more people stop to watch. The data suggests that when news outlets "spice up" their headlines, the internet eats it up, even if the spice makes the story less accurate.

Why This Matters

This paper is like giving us a microscope for the newsroom.

Before, we could only judge a news story by its final, polished version. We didn't know if the editor had to fight to keep a fact in, or if they decided to cut a detail to make the story punchier.

MediaSpin shows us that news isn't just "reported"; it is sculpted. Every time a word is added or removed, it changes how we see the world.

If we remove a country's name, that country becomes invisible.
If we add an emotional word, that event becomes scary.

The Bottom Line:
The next time you see a news headline, remember that it might not be the final version. It might have been tweaked, spiced up, or trimmed to fit a specific angle. This dataset helps us see those invisible hands shaping the news, reminding us to ask: "What did they change, and why?"

1. Problem Statement

While news framing and media bias are well-studied topics, prior research predominantly analyzes static, final published headlines. This approach overlooks the dynamic editorial process where headlines are revised post-publication. These revisions (edits) can subtly or significantly alter the framing of events by adding/removing emotional language, shifting agency, omitting context, or changing geographic specificity.

Research Gap: Existing datasets do not capture the revision history of news, making it difficult to study how subjective and objective biases are introduced or removed through specific editorial decisions. Furthermore, there is a lack of understanding regarding how these dynamic edits influence downstream audience engagement on social media.

2. Methodology

A. Data Collection: MediaSpin Dataset

Source: The authors built upon the NewsEdits dataset (Spangher et al., 2022), which contains 1.2 million articles from 22 outlets.
Selection: They curated 78,910 headline pairs (original vs. edited) from five English-language outlets representing a spectrum of political bias and credibility: Fox News, New York Times, Washington Post, Reuters, and Rebel.
Preprocessing: Headlines were cleaned (removing metadata like "| Fox News"), and word-level insertions and deletions were isolated.

B. Annotation Pipeline

Taxonomy: The authors defined 13 types of media bias, grounded in established taxonomies (Spinde et al., 2023; Hamborg et al., 2020), categorized into:
- Subjective Dimensions: Sensationalism, Spin, Mudslinging, Mind Reading, Opinion as Fact, Subjective Adjectives, Word Choice.
- Objective Dimensions: Unsubstantiated Claims, Slant, Flawed Logic, Omission, Omission of Source Attribution, Bias by Story Choice/Placement.
Automated Annotation: A Large Language Model (LLM) pipeline using GPT-3.5-turbo was employed to label the added/removed words for bias types.
Human Validation: Two co-authors independently reviewed 509 sampled instances.
- Inter-annotator Agreement: Overall 84.9% (Cohen's $\kappa$ = 0.67).
- Reliability: Subjective biases showed near-perfect agreement (e.g., Sensationalism 100%), while objective biases (requiring contextual inference) showed lower agreement (e.g., Flawed Logic 72.5%).
- Error Analysis: The LLM tended to be more cautious than humans regarding objective biases and occasionally misidentified the direction of bias (addition vs. removal).

C. Downstream Applications

The paper demonstrates the dataset's utility through three distinct analyses:

Cross-National Analysis: Using GeoNames and OpenStreetMap APIs, the authors tracked the addition/removal of country references to identify geographic framing asymmetries.
Bias Classification: Fine-tuning DeBERTa-v3-small transformers for binary (Subjective vs. Objective) and fine-grained (13-class) classification.
MediaSpin-in-the-Wild: A behavioral dataset linking the annotated headlines to 180,786 tweets from 819 consenting US users to analyze engagement metrics (likes, replies, retweets).

3. Key Results

A. Editorial Dynamics & Geographic Framing

Regional Asymmetry: The analysis revealed systematic patterns in how countries are framed.
- Added: The US, China, Iran, Germany, and Russia are most frequently added to headlines, foregrounding specific geopolitical actors.
- Removed: Belgium, the Philippines, Cuba, and Malaysia are frequently removed, often replaced by broader regional labels (e.g., "EU" instead of "Belgium"), effectively collapsing specific national contexts into regional narratives.
- Silencing: Certain regions (e.g., Palestinian Territories, Djibouti) are exclusively removed and never added, representing "representational silencing."

B. Machine Learning Performance

Binary Classification: The DeBERTa model achieved strong accuracy (~77.3%) and F1 scores for distinguishing between subjective and objective bias.
Fine-Grained Classification: Performance varied by category.
- High Performance: Subjective biases with clear lexical signals (e.g., Sensationalism, Spin) were detected reliably (F1 $\approx$ 0.68–0.71).
- Low Performance: Objective biases requiring inferential reasoning (e.g., Flawed Logic, Omission) were harder to detect (F1 $\approx$ 0.48–0.52), highlighting the challenge of detecting "missing" information.

C. Behavioral Impact (MediaSpin-in-the-Wild)

Engagement Amplification: Biased headlines generate significantly higher engagement than unbiased ones.
- Subjective Bias: Generated 2–3x more likes and 1.5–2x more replies.
- Objective Bias: Generated 2x more likes and 1.3–1.5x more retweets.
Implication: Both affective (emotional) and evidentiary (structural) biases drive viral spread, suggesting a socio-technical feedback loop where editorial choices that heighten cues (emotional or omission-based) increase visibility.

4. Key Contributions

MediaSpin Dataset: The first large-scale resource (78k pairs) capturing post-publication headline edits annotated for fine-grained media bias.
MediaSpin-in-the-Wild: A novel linkage of editorial bias annotations to real-world social media engagement data, bridging the gap between editorial intent and audience behavior.
Theoretical Insight: Demonstrates that framing is a dynamic process. The paper shows that "agenda-setting" (what is included) transitions into "framing" (how it is interpreted) through specific revision operations (additions vs. deletions).
Methodological Framework: Establishes a pipeline for detecting bias not just in static text, but in the micro-changes that shape public discourse.

5. Significance and Implications

Media Accountability: Provides a tool for watchdogs and researchers to audit editorial practices, revealing how specific populations or conflicts are systematically foregrounded or backgrounded.
Understanding Polarization: The findings suggest that both emotional framing and "epistemic distortions" (like omitting sources) contribute to the viral spread of news, complicating efforts to curb misinformation.
Limitations & Future Work:
- The dataset is currently limited to five English-language outlets, limiting generalizability to non-Western contexts.
- Objective bias detection remains challenging due to the need for external context.
- The engagement analysis is correlational; future work needs to control for topic salience and outlet reputation to isolate the causal effect of bias.

In conclusion, MediaSpin shifts the focus from static content analysis to process-oriented analysis, offering a reproducible benchmark to study how editorial revisions shape the information ecosystem and influence public discourse.