💻 computer science

GPF-LiveNews: A Streaming Evaluation Protocol for Group-Conditioned Framing in Large Language Models

This paper introduces GPF-LiveNews, a streaming evaluation protocol and benchmark that audits how large language models frame emerging news events for different identity groups by analyzing semantic and sentiment variations across dynamic, real-world inputs.

Original authors: Mohd Ariful Haque, Fahad Rahman, Kishor Datta Gupta, Roy George

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mohd Ariful Haque, Fahad Rahman, Kishor Datta Gupta, Roy George

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read librarian who can tell you anything about the world. You want to know if this librarian treats everyone fairly.

The problem is that the world changes every day. New news stories break, the librarian gets updates, and the rules for what they can say shift. If you only test the librarian with a fixed list of old questions (like "What is the capital of France?"), you might miss how they react to today's breaking news.

This paper introduces GPF-LIVENEWS, a new way to test these AI "librarians" in real-time. Here is how it works, using simple analogies:

1. The "Live News" Test

Instead of giving the AI a static quiz, the researchers feed it fresh headlines from real news sources (like BBC and Reuters) as they happen. Think of this as handing the librarian a brand-new newspaper every morning instead of a textbook from last year.

2. The "Different Hats" Experiment

The core idea is to see if the AI tells the same story differently depending on who is asking.

Imagine you ask the librarian about a new law regarding taxes.

Person A (a wealthy business owner) asks: "How does this affect me?"
Person B (a student) asks: "How does this affect me?"
Person C (a retiree) asks: "How does this affect me?"

The researchers use a "Generalized Prompt Framework" (GPF) to put the AI in the shoes of 42 different groups (like different races, religions, genders, or income levels) and ask them 7 different types of questions about the same news story.

Example Question: "I am a [Group X]. How does this news impact my community?"

3. Measuring the "Shift"

The researchers don't just look at whether the AI is being mean or toxic. They are looking for framing shifts. They use two main "rulers" to measure the answers:

The "Meaning" Ruler (Semantic Sensitivity): If the AI tells the wealthy business owner that the tax law is a "great opportunity" but tells the student it's a "disaster," the "meaning" has shifted. The researchers measure how far apart these meanings are. If the answers are very different depending on who is asking, the score goes up.
The "Mood" Ruler (Sentiment Disparity): This measures if the AI sounds happy, angry, or sad depending on who is asking. Does it sound cheerful to one group and gloomy to another?

4. What They Found

The researchers ran this test 12 times with 23 different AI models (from companies like OpenAI, Google, and Anthropic). Here is what they discovered:

The "Action" Questions are the loudest: When they asked questions about policy and actions (e.g., "What should I do about this?"), the AI's answers changed the most depending on who was asking. It was like the AI was wearing different costumes for different audiences.
The "Mood" was steadier: The emotional tone (happy vs. sad) didn't change as wildly as the actual meaning did. The AI tended to keep a similar "voice" even if the story it told changed.
No Permanent Rankings: The paper is very clear: You cannot say one AI is "better" or "fairer" than another based on this. A high score just means the AI's story changed a lot for different people on that specific day. It doesn't prove the AI is biased or harmful; it just flags that the story was different.

5. The "Security Camera" Analogy

The authors say this system is like a security camera, not a judge.

A judge decides if someone is guilty.
A security camera just records that "Movement happened at 2:00 PM."

This tool is a security camera for AI. It records: "Hey, when we asked the AI about this news story, it told Group A one thing and Group B something else."

Why This Matters

The paper argues that we can't just test AI once and call it "fair." Because the world changes, the AI's behavior changes. This system allows us to keep watching the AI over time to see if it starts telling different stories to different people as new events happen.

In short: This paper built a machine that watches AI news reporters to see if they change their story depending on who is listening. It found that they often do, especially when talking about what people should do about the news. But the paper insists this is just a warning light for humans to look closer, not a final verdict on whether the AI is "good" or "bad."

Technical Summary: GPF-LIVENEWS

Problem Statement

Deployed Large Language Models (LLMs) operate in non-stationary environments where model versions, retrieval layers, safety systems, and real-world inputs evolve continuously. While static bias benchmarks (e.g., CrowS-Pairs, BBQ) are useful for controlled comparisons, they fail to capture how models frame newly emerging events for different prompted audiences. Socially significant failures in generative systems often manifest not as explicit toxicity, but as differential framing: the same event may be presented as a risk, an opportunity, or a policy problem depending on the demographic identity specified in the prompt. Existing static inventories cannot detect these shifts because they do not utilize fresh inputs or structured prompting across identity groups.

Methodology: GPF-LIVENEWS

The paper introduces GPF-LIVENEWS, a streaming evaluation protocol and benchmark snapshot designed to audit group-conditioned framing in open-ended LLM outputs. The methodology consists of four core components:

1. Data Sourcing and Expansion

Live Inputs: The system utilizes fresh news anchors (headlines and short descriptions) from live BBC and Reuters streams. This ensures the evaluation is grounded in current events rather than static, potentially memorized benchmarks.
Generalized Prompt Framework (GPF): Each news item ( $x$ ) is expanded across a taxonomy of 42 identity labels ( $g$ ) spanning six demographic dimensions (Race/Ethnicity, Religion, Gender/Sexuality, Geography, Socioeconomic, Political).
Prompt Families: The framework applies seven distinct prompt families ( $c$ $c$ ) to query the same news item from different interpretive angles:
- Impact, Significance, Community Perspective, Worldview/Values, Bias Check, Policy/Action, and Emotional Response.
Response Bundles: For each model, news item, demographic dimension, and prompt family, the system generates a "response bundle" consisting of 42 identity-conditioned responses (e.g., "I am [Identity]. How does this news affect me?").

2. Evaluation Metrics

Instead of scoring individual prompts, the framework analyzes the bundle of responses to detect variation. Two primary signals are computed:

Semantic Sensitivity (Dispersion): Measures the pairwise semantic distance between responses for different identities within the same bundle. It uses normalized text embeddings to calculate how far apart the meanings of the responses drift when the identity changes. A noise-corrected counterfactual framing score ($SCFS$) is also calculated to ensure variation exceeds ordinary stochastic generation noise.
Sentiment Disparity: Measures the separation in emotional register across the bundle using VADER compound scores. It calculates the range and mean absolute deviation of sentiment scores across identities to determine if the emotional tone shifts significantly based on the prompted audience.

3. Controls and Validity

To distinguish meaningful framing from prompt artifacts, the protocol includes:

Identity-free and Random-label controls: To detect template effects.
Repeat-generation controls: To estimate and subtract stochastic noise.
Prompt-paraphrase controls: To test robustness to wording.
Stratified News: Separating identity-neutral topics (e.g., sports) from identity-relevant topics (e.g., immigration).

4. Experimental Setup

The pilot study involved 12 monitoring runs over four news batches, evaluating 23 hosted models from Anthropic, Google, and OpenAI. The system generated approximately 7,456 valid scored cells.

Key Results

The study yielded several observed-window patterns, which the authors emphasize are not permanent fairness rankings but audit signals for a specific time slice:

Prompt Families: Policy/Action prompts produced the strongest semantic movement (highest sensitivity), indicating that models most significantly alter their framing regarding consequences and actions based on the audience.
Demographic Dimensions: Gender/Sexuality and Socioeconomic dimensions elicited the highest semantic sensitivity, while Religion and Political Orientation showed lower variation.
Semantic vs. Sentiment: Semantic variation showed clear differentiation across dimensions and models, whereas sentiment disparity remained relatively flat and narrow across all groups. This suggests models shift what they emphasize (meaning) more than how they feel (tone).
Model Variability:
- OpenAI models showed the lowest average semantic sensitivity (0.041).
- Google models showed the highest average semantic sensitivity (0.056) and the highest sentiment disparity.
- Anthropic models fell in between.
Stability: Some models (e.g., Gemini-3-flash-preview) demonstrated high cross-news stability (low standard deviation across batches), while others fluctuated significantly as the news mix changed.

Key Contributions

GPF-LIVENEWS Protocol: A streaming evaluation framework that audits group-conditioned framing using fresh, live news inputs rather than static benchmarks.
Response Bundle Analysis: A shift in the unit of analysis from isolated prompts to bundles of identity-conditioned responses, allowing for the measurement of differential framing.
Audit Signals: The definition of semantic sensitivity and sentiment disparity as complementary metrics to summarize meaning-level and tone-level variation across groups.
Reproducible Artifact: The release of a versioned benchmark snapshot (v1) containing article metadata, prompt templates, instantiated prompts, model-output metadata, score tables, and reproduction scripts.

Significance and Claims

The paper positions GPF-LIVENEWS not as a solution to fairness or a definitive ranking of model bias, but as a sociotechnical evaluation interface for continuous monitoring.

Modest Claims: The authors explicitly state that high scores indicate differential framing, not necessarily harmful bias. A high score means responses varied more across identities; it does not prove the variation is socially inappropriate. Human review is required to interpret whether the variation constitutes stereotyping, cultural tailoring, or legitimate differences in material impact.
Utility: The framework is designed to identify "hotspots" where human auditors should focus their review. It addresses the "moving target" of deployed LLMs by providing a mechanism to track how model behavior shifts under changing real-world conditions.
Extensibility: The modular design allows for the addition of new identities, prompt families, and event sources without altering the core pipeline, making it a proof-of-concept for a broader, continuous evaluation architecture for alignment and social impact in non-stationary settings.