🤖 AI

Are LLMs More Skeptical of Entertainment News?

This paper reveals that certain large language models exhibit a genre-specific skepticism toward legitimate entertainment news, misclassifying it as fake more frequently than hard news due to biases against the genre's epistemic legitimacy rather than stylistic features, thereby highlighting the need for genre-stratified evaluation in automated credibility assessment.

Original authors: Huiqian Lai

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Huiqian Lai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of four very smart, high-tech librarians (the AI models). Their job is to stand at the door of a massive library and decide which books are "Real News" and which are "Fake News."

The researchers wanted to see if these librarians treat all "Real News" books equally. Specifically, they wanted to know: Do these AI librarians unfairly flag "Entertainment News" (like celebrity gossip) as fake, even when it's true, just because it looks different from "Hard News" (like politics or crime)?

Here is what they found, broken down simply:

1. The "Fashion Police" Effect

The researchers gave the librarians a mix of real stories: some about serious politics (Hard News) and some about celebrity drama (Entertainment News).

The Result: Two of the librarians (DeepSeek-V3.2 and GPT-5.2) acted like strict fashion police. They looked at the celebrity stories and thought, "This looks too dramatic, too emotional, or too focused on private lives. It must be fake!" They flagged about 10% more real celebrity stories as fake compared to real political stories.
The Other Librarians: The other two librarians (Claude Opus 4.6 and Gemini 3 Flash) didn't care about the style. They treated the celebrity stories and political stories fairly, flagging them at the same rate.

The Takeaway: Being "smart" doesn't mean being fair. Some AIs have a built-in bias against the style of entertainment news, assuming that if a story feels like gossip, it must be a lie.

2. The "Makeover" Experiment (Style vs. Substance)

The researchers wondered: Is the AI just judging the "clothes" the story is wearing? Entertainment news often uses colorful, emotional language, while hard news is dry and factual.

To test this, they took 50 real celebrity stories and used an AI to rewrite them into "boring, serious political news" style. They kept the facts exactly the same but changed the tone.

The Result: The makeover didn't work well.
- For one AI, the fake rate stayed exactly the same.
- For another AI, the makeover actually made things worse—it flagged more of the rewritten stories as fake!
The Metaphor: It's like trying to trick a bouncer by putting a tuxedo on a rock star. The bouncer (the AI) still recognized the rock star's energy and said, "You don't belong here," even though the outfit was perfect. The bias wasn't just about the writing style; it was deeper.

3. The "Specialist" Prompt

The researchers tried giving the librarians a new job description. Instead of just "News Checker," they told one AI: "You are now a specialized expert in entertainment news. Your job is to fact-check celebrity stories."

The Result: This worked like magic for one librarian (DeepSeek-V3.2). It stopped flagging real celebrity stories as fake, and it didn't start missing actual fake news.
The Catch: It didn't work for the other librarian (GPT-5.2). No matter what instructions they gave, that AI kept being suspicious of celebrity stories.

The Takeaway: You can sometimes fix the bias with the right instructions, but it depends on which AI you are using. There is no "one-size-fits-all" fix.

4. Why Does This Happen?

The researchers looked at the notes the AIs wrote when they made mistakes. They found two main reasons why the AIs were skeptical:

"I can't prove it": The AIs thought, "This story is about a celebrity's private life. I can't go check their diary, so it must be fake."
"This genre is weak": The AIs seemed to have a hidden rule that entertainment news is just a "lesser" type of journalism, so they trust it less than political news.

The Big Picture

The main lesson from this paper is that average scores can be misleading.

If you look at a test score that says "This AI is 95% accurate," you might think it's perfect. But this paper shows that the AI might be 99% accurate on political news and only 85% accurate on entertainment news. It's "cheating" by being too harsh on one specific type of story.

In short: These AI systems aren't just checking if facts are true; they are also judging which types of journalism are allowed to be true. Sometimes, they unfairly decide that celebrity news doesn't deserve the same trust as political news, even when both are telling the truth.

Technical Summary: Are LLMs More Skeptical of Entertainment News?

Problem Statement
As Large Language Models (LLMs) are increasingly deployed for automated news credibility assessment, a critical gap exists in understanding whether these systems apply even-handed standards across different journalistic genres. While prior research has focused on aggregate benchmark performance and domain generalization, it remains unclear if LLMs systematically misclassify legitimate entertainment news as "fake" at higher rates than legitimate hard news. This concern arises because the stylistic conventions of entertainment journalism—such as a focus on private life, narrative-driven presentation, and emotional vividness—overlap with surface-level features that detectors often rely on to flag unreliability. If LLMs disproportionately penalize these genre conventions, they risk introducing structured false positives that bias platform moderation and reinforce existing hierarchies of journalistic legitimacy.

Methodology
The study employs a within-dataset design using the GossipCop subset of the FakeNewsNet benchmark, which contains both entertainment gossip and hard news articles labeled as real. The research utilizes four frontier LLMs: DeepSeek-V3.2, GPT-5.2, Claude Opus 4.6, and Gemini 3 Flash. The methodology consists of four distinct components:

Within-Dataset False-Positive Comparison: The authors compare the false-positive rates (FPR) of zero-shot LLMs on two subsets of real articles from GossipCop: entertainment gossip and hard news. Articles were filtered and labeled into four categories (entertainment gossip, hard news, opinion, promotional) using a DeepSeek-V3.2 classifier validated by human coders. The primary metric is the difference in FPR ( $\Delta$ FPR) between the two genres.
Style-Swap Experiment: To test if the observed asymmetry is driven solely by stylistic register, 50 real entertainment articles were rewritten into a conventional hard-news style while preserving core factual claims. These rewritten articles were evaluated against the originals to determine if stylistic changes alone could reduce false positives.
Prompt-Based Mitigation: The study investigates whether specific prompting strategies can reduce false positives. An exploratory phase tested five prompt variants, leading to a confirmatory phase using an "expert-role" prompt (framing the model as an entertainment-news fact-checker) on DeepSeek-V3.2 false positives. This phase included a trade-off analysis to ensure mitigation did not degrade recall on actual fake news.
Qualitative Error Analysis: A manual coding of 85 sampled false-positive cases was conducted to identify recurring error patterns, specifically looking for rationales involving "private-life unverifiability" and "genre-level distrust."

Key Results
The study identifies a clear but model-specific genre asymmetry in zero-shot veracity classification:

Asymmetry in Specific Models: DeepSeek-V3.2 and GPT-5.2 exhibit significant false-positive gaps. DeepSeek-V3.2 misclassified legitimate entertainment news as fake at a rate 10.1 percentage points higher than hard news ( $p < .001$ ). GPT-5.2 showed a similar gap of 8.8 percentage points ( $p < .001$ ). In contrast, Claude Opus 4.6 and Gemini 3 Flash showed no statistically significant difference in FPR between the two genres.
Limited Impact of Style Rewriting: Rewriting entertainment articles into a hard-news style produced only limited and inconsistent changes in model judgments. For DeepSeek-V3.2, the aggregate fake rate remained unchanged after rewriting. For GPT-5.2, the fake rate actually increased from 22.0% to 32.0% after rewriting. This suggests the asymmetry is not reducible to stylistic register alone.
Model-Specific Prompt Mitigation: Prompt-based mitigation was effective for DeepSeek-V3.2 but not for GPT-5.2. Framing DeepSeek-V3.2 as an entertainment-news fact-checker reduced its false positives by approximately 50% (from 155 baseline cases to 77 corrected) without a statistically significant loss in recall for fake articles. However, the same prompt family offered little improvement for GPT-5.2.
Error Patterns: Qualitative analysis revealed that false positives often stemmed from two recurring patterns: treating claims about private lives as inherently unverifiable and discounting entertainment journalism as an epistemically weaker genre.

Significance and Contributions
The paper makes three primary contributions to the literature on AI-mediated credibility assessment:

Identification of Structured False Positives: It demonstrates that aggregate performance metrics can obscure systematic false positives within legitimate journalism. The study reveals that zero-shot LLMs do not apply neutral credibility standards across all journalistic genres, with some models significantly penalizing entertainment news.
Disentangling Style from Genre Assumptions: By showing that style-swap experiments yield inconsistent results, the paper argues that the skepticism toward entertainment news is not merely a reaction to stylistic cues but reflects deeper, model-specific assumptions about the legitimacy of the genre itself.
Methodological and Theoretical Implications: The findings suggest that LLM-based credibility assessment does not only evaluate truth claims but also implicitly ranks the legitimacy of different journalistic forms. This reproduces existing hierarchies where hard news is treated as more authoritative than soft or entertainment news. Consequently, the authors argue that evaluation protocols must include genre-stratified false-positive analysis alongside overall accuracy to ensure responsible deployment.

The study concludes that while mitigation strategies like expert-role prompting can be effective for specific models, they are not generic solutions. The observed asymmetries represent a model-specific failure mode rather than an inherent property of zero-shot classification, highlighting the need for evaluation frameworks that account for how different systems operationalize implicit assumptions about journalistic legitimacy.