💬 NLP

FactAppeal: Identifying Epistemic Factual Appeals in News Media

This paper introduces FactAppeal, a manually annotated dataset of 3,226 news sentences designed to advance the novel task of Epistemic Appeal Identification by capturing fine-grained evidentiary structures and source characteristics, alongside benchmarking encoder and generative models that achieve a macro-F1 score of 0.73.

Original authors: Guy Mor-Lan, Tamir Sheafer, Shaul R. Shenhav

Published 2026-03-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Guy Mor-Lan, Tamir Sheafer, Shaul R. Shenhav

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are reading a news headline that says, "The new bridge is safe to cross."

In the past, computers trying to fact-check this would just look at the sentence and ask, "Is this true or false?" They would treat the sentence like a standalone math problem.

But in real life, we don't just believe things because they are written down. We believe them because of who said them and how they said them.

Did a random person on the street say it?
Did the lead engineer who built the bridge say it?
Did a government report confirm it?
Did they show a photo of the bridge?

This paper introduces a new tool called FACTAPPEAL (which stands for "Factual Appeal") to teach computers how to understand these "trust signals."

The Core Idea: The "Credibility Detective"

Think of a news article as a courtroom.

The Claim: The lawyer (the journalist) makes a statement: "The defendant was at the scene."
The Evidence: The lawyer doesn't just shout it out; they point to a witness, a photo, or a police report.

Previous AI tools were like bad lawyers who only listened to the statement but ignored the evidence. FACTAPPEAL is a "Credibility Detective" that looks at the whole picture. It asks:

Is this actually a claim about reality? (Or is it just an opinion?)
If it is a claim, who is backing it up?
How are they backing it up? (Are they quoting a witness directly? Are they paraphrasing an expert? Are they showing a photo?)

The "Menu" of Trust

The researchers created a massive "menu" of different ways news sources try to convince us. They broke it down into two main categories, like a tree with two branches:

1. The "I Was There" Branch (Internal Sources)
These are people or things that were right at the scene of the action.

Active Participants: The people who did the thing (e.g., the politician who signed the law).
Witnesses: People who saw it happen but didn't do it (e.g., a bystander).
Direct Evidence: Physical proof found at the scene (e.g., a photo or a fingerprint).

2. The "I Know About It" Branch (External Sources)
These are people who weren't there but have special knowledge or authority.

Experts: Scientists or specialists who know the theory behind the event.
Officials: People with legal or political power (e.g., the EPA director).
Documents/Reports: Studies or previous news reports that summarize the facts.

The Dataset: A Giant Library of Annotated Sentences

To teach the computer, the authors built a library called FACTAPPEAL.

They took 3,226 sentences from real news articles.
Human experts went through them like a team of editors, highlighting exactly which words were the claim and which words were the source.
They even tagged how the source was mentioned (e.g., "Named" vs. "Unnamed," "Direct Quote" vs. "Paraphrased").

It's like giving a student a textbook where every single fact is color-coded to show exactly where the information came from.

The Experiment: Teaching the Robot

The researchers tried to teach different types of AI models to read these sentences and find the "trust signals."

They tried Encoder models (think of these as careful readers who analyze words one by one).
They tried Generative models (think of these as creative writers who understand the whole story at once).

The Result: The best model (a large AI called Gemma 2) got about 73% accuracy. This is a good start, but it shows that understanding why we trust a news story is still a very hard puzzle for computers. The AI is getting better at spotting the "who" and "how," but it still struggles with the most subtle clues.

Why Does This Matter?

In a world full of fake news and confusion, knowing how a fact is supported is just as important as the fact itself.

For Fact-Checkers: Instead of just checking if a claim is true, they can see if the claim is backed by a credible expert or just a rumor.
For Society: It helps us understand how media shapes our reality. Are we being told things because of hard evidence, or because a politician is shouting?
For the Future: This tool helps us build AI that doesn't just "read" the news, but actually "understands" the trustworthiness of the news.

In short: FACTAPPEAL is teaching computers to stop taking news at face value and start asking, "Who told you that, and why should we believe them?"

1. Problem Definition

The paper addresses a critical gap in Natural Language Processing (NLP) regarding factual claim verification. While existing research excels at detecting claims and verifying their truthfulness against evidence (e.g., FEVER, SciFact), it largely overlooks the epistemic structures that confer credibility to those claims.

In news media, a claim's persuasiveness often relies not just on its content, but on how it is anchored to external sources (e.g., experts, witnesses, official reports). The authors propose a new task: Epistemic Appeal Identification. This task requires:

Determining if a sentence contains a factual claim.
Identifying if the claim is supported by an external source (an epistemic appeal).
Classifying the type of source (e.g., Expert, Witness, Official) and the method of appeal (e.g., Direct Quote, Indirect Paraphrase).

2. Methodology

A. Dataset Construction (FACTAPPEAL)

The authors created FACTAPPEAL, a manually annotated dataset consisting of 3,226 English-language news sentences (published 2020–2022).

Annotation Scheme: The dataset uses span-level annotations (character indices) rather than just sentence-level labels. This allows for the differentiation of factual claims, non-factual statements, and the specific sources supporting them within a single sentence.
Tagging Taxonomy:
- Factuality: Distinguishes between Fact Without Appeal (bare facts) and Fact With Appeal (facts supported by a source).
- Source Types: Categorized based on proximity to the event (Internal vs. External) and nature (Human vs. Non-human). Specific classes include:
  - Internal: Active Participant, Witness, Official, Direct Evidence.
  - External: Expert, Expert Document, News Report.
- Modifiers: Annotations capture whether the source is Named/Unnamed, the Quotation Type (Direct vs. Indirect), and attributes like Recipient, Time, and Location of the appeal.
Inter-Annotator Agreement (IAA): The dataset achieved substantial agreement with an overall IoU of 0.74 and Cohen's Kappa of 0.82.

B. Modeling Approaches

The authors evaluated two distinct modeling strategies on the test set (15% of the data):

Encoder Models (Token-level Classification):
- Treated as a multi-label binary classification problem at the token level.
- Models: Fine-tuned RoBERTa, DeBERTa v3, and ModernBERT (base versions).
- Loss: Focal Loss to handle class imbalance.
Generative Decoder Models (Sequence-to-Sequence):
- Treated as a text generation task where the model outputs the raw sentence with XML-style tags inserted.
- Models: Fine-tuned Gemma 2 (2B, 9B), Llama 3.1 (8B), and Mistral v0.3 (7B).
- Training: Used QLoRA with 4-bit quantization.

3. Key Results

Best Performance: The Gemma 2 9B generative model achieved the highest macro-F1 score of 0.73.
Encoder vs. Decoder: While top encoder models (RoBERTa) performed competitively (F1 ~0.70), generative models generally showed more robustness, particularly in handling the complex nested structure of the tags.
Granular Performance:
- High Performance: Models performed well on general "Factuality" detection (F1 > 0.85) and distinguishing between Named/Unnamed sources.
- Challenging Areas: Performance dropped significantly for rare source types (e.g., "Witness," "Direct Evidence") and specific attributes like "Appeal Location."
- Correlation: Encoder models showed a stronger correlation between tag frequency and F1 score ( $\rho_s = 0.72$ ) compared to decoders ( $\rho_s = 0.66$ ), suggesting decoders generalize better to rare classes.

4. Key Contributions

Novel Task Formulation: Introduced Epistemic Appeal Identification, shifting focus from what is claimed to how credibility is constructed via sources.
FACTAPPEAL Dataset: Provided the first large-scale, span-level annotated resource that captures the nuanced relationship between factual claims and their evidentiary bases (source type, attribution method, credentials).
Comprehensive Taxonomy: Developed a structured typology of epistemic sources distinguishing between internal (direct connection) and external (expertise-based) authorities.
Empirical Benchmarking: Established a baseline for this task using both encoder and generative architectures, demonstrating the feasibility of sequence-to-sequence approaches for complex epistemic modeling.

5. Significance and Implications

For Automated Fact-Checking: The dataset enables systems to move beyond simple claim verification to analyzing the structure of justification. Understanding who is cited and how can improve the detection of misinformation that relies on fake or misattributed authority.
For Discourse Analysis: The tool allows researchers to quantitatively study how media constructs reality, identifying patterns in how different political or social groups rely on specific types of epistemic authority (e.g., experts vs. witnesses).
Social Science Impact: By systematically modeling epistemic appeals, the work provides a computational framework to investigate "deep disagreements" in political discourse and the role of media in shaping public perception of facts.
Limitations & Future Work: The current scope is limited to English sentences. Future work aims to extend annotations to paragraph/article levels to model complex discourse structures and to apply the schema to multilingual and social media contexts.

In summary, FACTAPPEAL represents a significant step forward in NLP by treating the "source of truth" not just as a binary label, but as a rich, structured element of discourse that can be computationally modeled to better understand media credibility and misinformation.