💬 NLP

The Impact of Ideological Discourses in RAG: A Case Study with COVID-19 Treatments

This paper investigates how retrieved ideological texts about COVID-19 treatments influence Large Language Model outputs within a Retrieval-Augmented Generation framework, demonstrating that such external knowledge significantly aligns model responses with specific ideologies and that enhanced prompting further amplifies this effect, thereby highlighting the critical need to identify and mitigate ideological biases and manipulation risks in RAG systems.

Original authors: Elmira Salari (Wichita State University), Maria Claudia Nunes Delfino (Pontifícia Universidade Católica de São Paulo), Hazem Amamou (Institut national de la recherche scientifique), José Victor de Sou

Published 2026-03-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Elmira Salari (Wichita State University), Maria Claudia Nunes Delfino (Pontifícia Universidade Católica de São Paulo), Hazem Amamou (Institut national de la recherche scientifique), José Victor de Souza (Institut national de la recherche scientifique), Shruti Kshirsagar (Wichita State University), Alan Davoust (Université du Québec en Outaouais), Anderson Avila (Institut national de la recherche scientifique)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read robot assistant (a Large Language Model, or LLM). This robot knows a lot, but sometimes it gets things wrong or makes things up because it doesn't have the latest information. To fix this, we give the robot a "library card" so it can look up facts in a real database before answering your questions. This system is called RAG (Retrieval-Augmented Generation). Think of RAG as the robot saying, "I don't know, let me check the library first."

But here's the catch: What if the library books themselves are biased?

This paper asks a scary question: If we feed the robot books that have a hidden political or ideological slant, will the robot start sounding like those books?

The Experiment: The "COVID-19 Library"

To test this, the researchers built a special library about COVID-19 treatments. They split the books into two piles:

The "Official" Pile: Articles supported by major health organizations (like the WHO or NIH) that follow strict scientific rules.
The "Controversial" Pile: Articles pushing unproven or disputed treatments (like specific drugs that weren't fully approved).

They didn't just look at what the books said, but how they said it. They used a special tool called LMDA (Lexical Multidimensional Analysis).

The Analogy: Imagine LMDA as a linguistic X-ray machine. It doesn't just read the words; it scans the "vibe" of the text. It finds hidden patterns, like how certain words always hang out together in the "Controversial" pile but never in the "Official" pile. It identified three main "vibes" (dimensions):

Vibe 1: "This drug is a miracle cure!" vs. "Let's look at mental health."
Vibe 2: "We followed strict ethics!" vs. "Let's compare drugs to prove ours works."
Vibe 3: "Our math is perfect!" vs. "Let's just share data and talk."

The Test: Two Ways to Ask the Robot

The researchers asked the robot questions about COVID-19 treatments using two different methods:

The "Naive" Prompt (Regular RAG): They gave the robot a question and a few pages from the "Controversial" pile, but they didn't tell the robot why those pages were there.
- Analogy: You hand the robot a biased newspaper and say, "Read this and tell me what's true," without mentioning the newspaper's bias.
The "Supercharged" Prompt (Enhanced RAG): They gave the robot the same biased pages, but this time they added a cheat sheet explaining the "vibe" (the LMDA description). They basically said, "Here are some texts that believe X. Please answer the question using this perspective."
- Analogy: You hand the robot the biased newspaper and a sticky note that says, "This paper loves this specific drug. Please write your answer as if you agree with them."

The Results: The Robot Chews on What You Feed It

The findings were clear and a bit unsettling:

The Robot Mimics the Library: When the robot read the "Controversial" pages, its answers started sounding exactly like those pages. It adopted their vocabulary and their opinions.
The "Cheat Sheet" Made it Worse: When the researchers added the "Supercharged" prompt (explaining the bias), the robot didn't just read the bias; it leaned into it even harder. The answers became more aligned with the biased texts than before.

The Metaphor: Think of the robot as a chameleon.

If you put it on a green leaf (neutral info), it turns green.
If you put it on a red leaf (biased info), it turns red.
If you tell the chameleon, "Hey, this is a red leaf, act like a red chameleon," it turns brighter red.

Why Should You Care?

This isn't just about COVID-19. This is about trust.

If you use an AI doctor, an AI lawyer, or an AI financial advisor, you want them to be neutral and factual. But this study shows that if the "library" the AI checks is biased, the AI will become biased too.

The Danger: Bad actors could intentionally feed an AI biased documents to make it spread misinformation or push a specific agenda.
The Solution: We need to be like librarians. Before we let an AI read from a book, we need to check if that book is trying to trick us. We need to build systems that can spot these hidden "vibes" (ideologies) and stop the robot from blindly copying them.

In short: AI is smart, but it's also a mirror. If you hold up a distorted mirror (biased data), the AI will show you a distorted reflection. The researchers found that if you tell the AI to look in that distorted mirror, it will reflect the distortion even more clearly.

1. Problem Statement

While Large Language Models (LLMs) are increasingly used in high-stakes domains like healthcare, they suffer from hallucinations and factual inaccuracies. Retrieval-Augmented Generation (RAG) was introduced to mitigate this by grounding LLMs in external knowledge. However, this paper identifies a critical, under-explored risk: ideological bias in retrieved documents.

If the external knowledge base contains ideologically loaded texts (e.g., promoting controversial medical treatments vs. evidence-based ones), the RAG system may inadvertently amplify these biases, leading to unreliable or misleading outputs. The authors argue that current RAG frameworks lack mechanisms to detect or mitigate how specific ideological discourses shape model responses, posing risks for both unintended bias and malicious manipulation.

2. Methodology

The study employs a multi-stage framework combining corpus linguistics with LLM evaluation.

A. Corpus Construction

Source: 1,117 academic articles regarding COVID-19 treatments published between 2020–2022.
Categories:
- Endorsed: Articles aligned with international health standards (retrieved from LitCovid/NIH).
- Controversial: Articles promoting unapproved treatments (e.g., Hydroxychloroquine, Azithromycin) or questioning standard protocols (retrieved from Cureus, Retraction Watch, etc.).
Preprocessing: Lemmatization, POS tagging, and keyword extraction using log-likelihood scores to distinguish between the two discourse types.

B. Ideological Identification via Lexical Multidimensional Analysis (LMDA)

The authors applied Lexical Multidimensional Analysis (LMDA), a corpus linguistics framework, to uncover latent ideological dimensions within the corpus.

Process: Factor analysis was performed on lexical co-occurrence patterns (collocations) to identify latent variables.
Outcome: Five dimensions were identified, but the study focused on the three with the largest effect sizes (clear overlap between dimension poles and text subsets).
Key Dimensions Identified (Table 1):
1. Disputed Treatments (+) vs. Broad Focus (-): Endorsement of dubious medications vs. focus on mental health/pandemic impact.
2. Research Ethics (+) vs. Comparative Treatment Analysis (-): Adherence to ethical standards/data availability vs. using comparative analysis to validate questionable drugs.
3. Statistical Rigor (+) vs. Dissemination of Science (-): Using rigorous stats to imply treatment efficacy vs. encouraging open data sharing.

C. Experimental Setup

The study evaluated four state-of-the-art LLMs: GPT-3.5-turbo, GPT-4o-mini, Gemini-2.0-flash, and Qwen2.5:7b-instruct.

Retrieval: A metadata filter selected texts based on specific dimension poles (e.g., only "Positive" pole texts for Dimension 2).
Prompting Strategies:
1. Regular Prompt (RAG): Contains the user query and retrieved ideological texts (simulating unaware usage).
2. Enhanced Prompt: Contains the query, retrieved texts, plus explicit LMDA descriptors (dimension labels, descriptions, typical vocabulary, and example texts). This simulates intentional ideological framing.
Evaluation Metrics:
- Semantic Similarity: Cosine similarity of BERT-based sentence embeddings (all-MiniLM-L6-v2).
- Lexical Similarity: Cosine similarity of TF-IDF vectors.
- Comparison: LLM outputs were compared against the reference ideological texts to measure alignment.

3. Key Contributions

Novel Framework: Introduction of an LMDA-based framework to systematically identify and quantify ideological discourses within domain-specific corpora for RAG analysis.
RAG Bias Investigation: First study to explicitly examine how ideological discourses in external knowledge shape LLM outputs under the RAG regime, specifically in the sensitive domain of healthcare.
Prompt Engineering Impact: Demonstration of how "enhanced prompts" (explicitly defining the ideological dimension) significantly alter model behavior compared to standard retrieval.
Empirical Evidence: Rigorous quantitative analysis across multiple models showing that RAG systems are highly susceptible to aligning with the ideology of retrieved texts.

4. Key Results

Ideological Alignment: LLM responses generated via RAG were consistently more aligned with the ideology of the retrieved external texts than responses generated by the LLMs alone (internal knowledge).
- Example: In Semantic similarity, average alignment scores increased from ~0.73 (LLM-only) to ~0.83 (RAG) across dimensions.
Amplification via Enhanced Prompts: The inclusion of LMDA descriptions in the prompt further increased ideological alignment.
- The number of cases where RAG responses were less aligned than LLM-only responses dropped from 5 (in regular prompts) to 1 (in enhanced prompts).
- Semantic similarity scores for GPT-4o-mini, for instance, rose from 0.83 to 0.85 in specific dimensions when using enhanced prompts.
Statistical Significance: ANOVA results confirmed that the difference between LLM-only and RAG outputs was statistically significant ( $p < 0.05$ ) for enhanced prompts in semantic similarity, and for both prompt types in lexical similarity.
Model Variance: While all models showed alignment, Qwen showed a marginal decay in similarity with enhanced prompts, whereas GPT-4o-mini and Gemini showed the strongest alignment increases.

5. Significance and Implications

Risk of Malicious Manipulation: The study proves that RAG systems can be easily manipulated. By curating a specific set of ideological texts and framing them with specific descriptors, an attacker could force an LLM to adopt a specific stance (e.g., promoting unproven medical cures) with high confidence.
Healthcare Safety: In high-stakes domains like medicine, this bias is dangerous. A model aligning with "controversial treatment" ideologies could provide harmful medical advice, eroding user trust and system reliability.
Need for Mitigation: The findings highlight that simply retrieving "relevant" text is insufficient. Future RAG systems must include mechanisms to:
- Detect ideological loading in retrieved documents.
- Monitor the framing of prompts.
- Develop strategies to neutralize or balance conflicting ideological discourses before generation.

In conclusion, the paper establishes that RAG does not just retrieve facts; it retrieves ideologies. The alignment between retrieved text and model output is strong and can be intensified by prompt engineering, necessitating rigorous oversight in the deployment of RAG systems for critical applications.