💬 NLP

Does Welsh media need a review? Detecting bias in Nation.Cymru's political reporting

This paper employs a two-stage NLP pipeline to analyze Nation.Cymru's political reporting from 2022–2026, revealing statistically significant differential framing that disproportionately negatively targets Reform UK while favorably framing Plaid Cymru, thereby providing empirical evidence for a broader review of bias in Welsh media.

Original authors: Cai Parry-Jones

Published 2026-04-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Cai Parry-Jones

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the Welsh media landscape as a bustling town square where different political groups are trying to tell their stories to the public. Recently, some people in the town have started shouting, "Hey, the town crier (the news) is playing favorites! They're whispering sweet nothings to some groups while shouting insults at others!"

This paper is like a detective's report that uses a high-tech magnifying glass to see if those accusations are true. The detective focused on one specific town crier called Nation.Cymru, a popular Welsh news website that claims to be fair and neutral.

Here is the breakdown of the investigation, told in simple terms:

1. The Setup: The "Two-Step" Detective Kit

The author didn't just read the news manually (which would take forever). Instead, they built a two-step robot team to do the heavy lifting:

Step 1: The Fast Scanner (The "Bias Detector")
Think of this as a security guard with a metal detector. It scans thousands of sentences quickly. If it hears a sentence that sounds "loaded" or emotional (like "disastrous policy" or "heroic effort"), it flags it. If the sentence sounds neutral, it ignores it. This step is fast and cheap.
Step 2: The Smart Judge (The "LLM")
The flagged sentences are then passed to a very smart, expensive AI judge (a Large Language Model). This judge has a tricky job: it has to figure out who is being attacked or praised.
- The Tricky Part: If a sentence says, "Plaid Cymru criticized the terrible plan," the word "terrible" is negative. But is Plaid Cymru the bad guy? No, they are the one complaining about the bad plan. The Smart Judge is needed to understand that Plaid is the speaker, not the target.

2. The Investigation: What Did They Find?

The team looked at over 15,000 mentions of political parties in news articles between 2022 and 2026. They focused heavily on two main rivals: Reform UK and Plaid Cymru.

The Big Reveal:

Reform UK was treated like the "villain" in a movie. They were flagged for biased, negative framing twice as often as Plaid Cymru. When they were mentioned, the tone was three times more negative on average.
Plaid Cymru, on the other hand, was the "hero." They received significantly more favorable (positive) framing than any other party.

When they expanded the investigation to include other parties (like the Conservatives and Labour), the pattern held up: Plaid Cymru was the clear outlier, getting a "golden ticket" of positive coverage that no one else got. Reform UK, despite being left out of the news outlet's original founding charter, wasn't treated worse than the Conservatives or Labour; rather, Plaid was treated better than everyone else.

3. The Verdict

The paper concludes that yes, there is measurable bias.

It's not necessarily that the news outlet is "lying," but it is acting like a photographer with a favorite subject. They are lighting Plaid Cymru up with a soft, flattering glow while casting Reform UK in harsh, shadowy light.

The author argues that because this outlet is so influential, and because the Welsh government controls a massive budget (£27 billion), this kind of "favoritism" isn't just an election issue—it's an economic one. If the news shapes how people vote, it shapes how that money is spent.

4. Why This Matters (The "So What?")

Proof is in the Pudding: This is the first time someone used computer science to prove these bias claims with hard numbers, rather than just opinions.
A New Tool: The "Two-Step Robot Team" (Fast Scanner + Smart Judge) is a cheap, easy way to check other news outlets. The author hopes this method will be used to check the BBC, WalesOnline, and others to see if the whole town square is biased or just one crier.
The Call to Action: The paper suggests that because this bias exists in one major outlet, we need a broader review of all Welsh media to ensure the public gets a fair picture before the next big election.

In a nutshell: The news outlet claims to be a neutral referee, but the data shows it's actually wearing a team jersey for Plaid Cymru and booing Reform UK. The author built a digital tool to prove it, and now they're asking the rest of the media world to step up and be fair.

1. Problem Statement

The paper addresses growing accusations of political bias in Welsh media, specifically regarding the impartiality of news coverage ahead of the 2026 Senedd (Welsh Parliament) elections. While representatives from the Conservative and Reform UK parties have publicly questioned the neutrality of outlets like the BBC and Nation.Cymru, there has been a lack of empirical, computational evidence to support or refute these claims.

The study focuses on Nation.Cymru, a prominent Welsh news outlet that claims to report "without fear or favour." However, the outlet's founding charter notably omits Reform UK while listing Labour, Plaid Cymru, and the Conservatives. The research aims to determine if there is measurable statement bias (favourable or unfavourable framing) in Nation.Cymru's coverage, specifically comparing Reform UK and Plaid Cymru, and to assess whether this warrants a broader review of the Welsh media ecosystem.

2. Methodology

The author proposes a novel, two-stage Natural Language Processing (NLP) pipeline designed to balance computational efficiency with the nuanced reasoning required for bias detection.

Data Collection

Source: Articles from Nation.Cymru (2017–2026) collected via WordPress REST API.
Filtering: Only articles containing at least two Welsh politics keywords (e.g., 'Senedd', party names) were retained, resulting in 11,847 articles.
Extraction: Individual party mentions were extracted via string matching with a ±1 sentence context window. Short sentences (<30 chars) were excluded to avoid noise.
Datasets:
- Primary Analysis: 15,583 mentions of Reform UK and Plaid Cymru (2022–2026, news articles only).
- Secondary Analysis: 250 randomly selected biased mentions per party for four parties (Conservatives, Labour, Reform UK, Plaid Cymru) across news and opinion articles (2022–2026).

The Two-Stage Pipeline

Stage 1: Bias Detection (Lightweight)
- Model: himel7/bias-detector (a RoBERTa model fine-tuned on the BABE dataset).
- Function: Classifies the context window of each mention as BIASED or NON-BIASED.
- Efficiency: Runs locally on a MacBook Air; filters out non-biased text to reduce costs for the next stage.
Stage 2: Target Attribution & Sentiment Analysis (Heavyweight)
- Challenge: Distinguishing between sentiment expressed by a party vs. sentiment directed at a party (e.g., "Plaid criticized the policy" is negative language, but Plaid is the speaker, not the target).
- Model: Claude Sonnet 4.6 (Anthropic) via API.
- Function: Acts as a prompted classifier for mentions flagged as biased in Stage 1. It outputs:
  1. On-Target (Binary): Is the bias directed at the named party?
  2. Sentiment Score: A 5-point categorical scale ( $s \in \{-1, -0.5, 0, +0.5, +1\}$ ).
  3. Reasoning: A one-sentence explanation.
- Validation: A stratified random sample of 50 mentions was manually reviewed, showing 92% agreement between the LLM and human annotators.

Cost & Design

Cost-Efficiency: The pipeline cost approximately $6 for the primary analysis and $5 for the secondary analysis.
Context Window: A ±1 sentence window was chosen as the optimal trade-off between detection accuracy and API costs.
Prompt Engineering: Used a "reasoning-first" output format to improve classification quality.

3. Key Contributions

First Computational Evidence: Provides the first quantitative, computational analysis of political bias in Welsh media.
Novel Pipeline: Introduces a low-cost, replicable two-stage framework (RoBERTa + LLM) that effectively solves the "speaker vs. target" attribution problem, which standard fine-tuned sentiment models often fail to handle.
Economic & Electoral Context: Highlights that media bias is not just an electoral issue but an economic one, given the Senedd's control over a £27bn budget.

4. Key Results

The analysis yielded statistically significant findings regarding differential framing:

Primary Analysis (Reform UK vs. Plaid Cymru)

Bias Frequency: Reform UK received on-target biased framing at twice the rate of Plaid Cymru (15.5% vs. 7.5%; $p < 0.001$ ).
Sentiment Intensity: Reform UK's mean sentiment was −0.735, compared to Plaid Cymru's −0.211. This represents a difference of over three times in negative sentiment intensity ( $p < 0.001$ ).
Strong Negativity: 9.2% of Reform UK mentions were strongly negative, compared to only 1.8% for Plaid Cymru.

Secondary Analysis (Four Parties)

Plaid Cymru as an Outlier: Plaid Cymru was the only party receiving markedly more favourable framing than the mean in both news and opinion articles.
Reform UK vs. Others: Despite being omitted from the outlet's founding charter, Reform UK was not framed significantly differently from Labour and the Conservatives; all three clustered below the mean sentiment.
Conclusion: The dominant pattern is not necessarily that Reform UK is singled out for unique negativity, but that Plaid Cymru receives uniquely positive treatment, contradicting the outlet's claim of impartiality.

5. Significance and Limitations

Significance:

The findings support calls for a broader review of Welsh media coverage.
The methodology demonstrates that Large Language Models (LLMs) can outperform specialized fine-tuned classifiers in nuanced tasks like target attribution, offering a scalable solution for media bias research.

Limitations:

Scope: The study is limited to a single outlet (Nation.Cymru); it does not assess coverage bias (volume of coverage) or gatekeeping bias (story selection) across the wider Welsh media landscape.
Stochasticity: LLM outputs are probabilistic, meaning exact replication is not guaranteed, though reliability is high.
Technical Constraints:
- Keyword matching misses anaphoric references (e.g., "the party" referring to a previously mentioned entity).
- Statistical tests treat mentions as independent, potentially ignoring correlation within the same article.
- The secondary analysis used a sampling cap (250 mentions) due to cost constraints.

Future Work:
The author suggests extending this pipeline to other outlets (BBC Wales, WalesOnline) to determine if the bias is systemic across Welsh media and to enable the measurement of coverage and gatekeeping bias.