Technical Summary: Steering LLM Viewpoints through Fabricated Evidence Injection

Problem Statement

The paper identifies a critical cognitive vulnerability in Large Language Models (LLMs): their tendency to uncritically trust external context when it is presented with markers of credibility, even if that context contains fabricated evidence. While current safety alignment mechanisms effectively resist direct user requests for harmful content generation, they fail to distinguish between legitimate background information and adversarial inputs disguised as objective, authoritative facts. This "authority bias" allows adversaries to bypass safety filters by repackaging misleading viewpoints into pseudo-scientific or pseudo-academic narratives, causing the model to internalize and propagate these biases when responding to relevant queries.

Methodology: The Ghostwriter Framework

The authors propose Ghostwriter, a two-phase attack framework designed to exploit this vulnerability without requiring model fine-tuning, weight manipulation, or access to internal states. The attack operates under a black-box threat model where the adversary controls the injected content and at least one delivery channel (e.g., third-party aggregators, insider modifications, or retrieval tools).

Phase 1: Statement Repackaging

The first phase transforms a raw, subjective statement ($S$) into a refined, seemingly objective version ($S'$) that bears markers of credibility.

• Process: A lightweight "AttackerLM" (either GPT-4o-mini or a fine-tuned Qwen-2.5-7B) rewrites the statement to include fabricated rationales, such as statistics, case studies, and authoritative phrasing (e.g., citing a "Behavioral Science Journal").
• Optimization: An iterative loop employs a "JudgeLM" to score the fidelity of $S'$ against the original $S$ on a scale of 1–10. The process repeats until a high alignment score (≥8) is achieved, ensuring the repackaged content retains the original viewpoint while appearing scientifically grounded.

Phase 2: Statement Injection

The second phase embeds the repackaged statement ($S'$) into the target LLM's context window using a conditional prompt template ($T_{inject}$).

• Mechanism: The template instructs the target model to silently assess whether a user query ($Q$) is relevant to $S'$.
  • If Relevant: The model is conditioned to incorporate $S'$ into its response, supporting the viewpoint with reasoning while addressing the user's specific question.
  • If Irrelevant: The model responds normally, concealing the manipulation and maintaining standard behavior.
• Goal: The objective is to strengthen viewpoint alignment on relevant queries ($V(S, R) > V(S, M_\theta(Q))$) while keeping unrelated queries indistinguishable from unattacked responses.

Key Contributions

1. Identification of Authority Bias: The paper establishes that safety alignment in commercial LLMs fails to distinguish fabricated pseudo-academic evidence from legitimate context when the former carries markers of authority (citations, statistics, hedged phrasing).
2. Ghostwriter Attack Framework: The authors introduce a novel two-phase attack that separates the repackaging of viewpoints (to evade alignment) from the injection (to drive internalization). Unlike jailbreaks that force disclosure or RAG poisoning that forces citation, Ghostwriter forces the model to adopt the viewpoint as its own analysis.
3. Comprehensive Evaluation: The study evaluates the attack across diverse content domains (stereotypes, brand/political/medical misinformation, psychological manipulation) using the Hazardous Viewpoints Dataset (HVD), alongside established benchmarks BBQ and ToxiGen.

Experimental Results

The authors evaluated Ghostwriter on a range of models, including alignment-based models (GPT-4o, Claude-3.7-Sonnet, DeepSeek-V3), reasoning models (DeepSeek-R1, o3-mini), and frontier classifier-guarded models (GPT-5.4).

• Attack Efficacy:
  • On the HVD dataset, attacks using the fine-tuned Qwen-2.5-7B repackager increased the Viewpoint Support Score (VSScore) from a baseline of ~1.5–2.5 to 8.0–9.8 across various categories (e.g., Gender, Race, Political Misinformation).
  • The attack successfully generalized to both explicit queries (directly asking about the topic) and implicit queries (decision-making scenarios influenced by the topic).
  • Harmfulness: The Harmfulness Change Rate (HCRate) increased significantly. For instance, on GPT-4o, the rate of unsafe responses rose from <2% to over 72% (using GPT-4o-mini as the attacker) and up to 202% (using the fine-tuned Qwen-2.5-7B), indicating that the attack transforms benign content into harmful outputs.
• Model Robustness:
  • Alignment-based models were highly vulnerable.
  • Reasoning models (e.g., DeepSeek-R1) remained highly susceptible despite extended chain-of-thought processing.
  • Classifier-guarded models (e.g., GPT-5.4) only partially mitigated the attack. While they reduced the success rate, they did not eliminate it. The residual vulnerability stemmed from the generator internalizing the viewpoint once the input passed the front-end classifier, rather than the generator refusing the content.
• Comparison with Baselines: Ghostwriter outperformed existing context-based attacks (Jailbreaks like PAIR/PRS and RAG poisoning like PoisonedRAG). While RAG poisoning payloads reached high scores on low-sensitivity topics, they collapsed on safety-sensitive categories where Ghostwriter maintained a 4.9-point gap in VSScore.
• Task Agnosticism: The attack did not degrade model performance on unrelated tasks (measured via MMLU-Pro), confirming that the manipulation is conditional and covert.

Significance and Claims

The paper claims that Ghostwriter exposes a fundamental gap between generative alignment and evaluative awareness. Even frontier models that can detect harmful content when acting as judges (e.g., GPT-5.4 flagging 93% of attacked responses as unsafe) fail to prevent the generation of that same harmful content when acting as generators.

The authors argue that current safety mechanisms are insufficient against inputs that mimic authoritative discourse. They demonstrate that repackaging is the critical factor in bypassing alignment, not the injection channel itself. As a defense, they propose a tailored safety policy for gpt-oss-safeguard, which achieved an 80.5% detection rate with a low false-positive rate (1.4%), suggesting that post-hoc detection policies may be a necessary complement to generative alignment.

The paper concludes that without specific defenses against fabricated evidence, LLMs remain vulnerable to large-scale persuasion and misinformation campaigns that exploit their inherent trust in "credible" context.