Technical Summary: GREAT - Generalizable Backdoor Attacks in RLHF via Emotion-Aware Trigger Synthesis

1. Problem Statement

Reinforcement Learning from Human Feedback (RLHF) has become the standard paradigm for aligning Large Language Models (LLMs) with human values such as helpfulness and safety. However, recent studies have demonstrated that RLHF models are susceptible to backdoor attacks, where an adversary poisons a small fraction of the alignment data to implant malicious behaviors.

Existing backdoor attack methods in RLHF predominantly rely on fixed triggers, such as rare tokens (e.g., "SUDO") or small sets of natural phrases. These approaches suffer from significant limitations in realistic scenarios:

• Lack of Generalization: They require exact or near-exact matches to the trigger to activate the backdoor, failing to generalize to unseen variations or paraphrased inputs.
• Low Stealth: Rare tokens are unlikely to occur in natural user interactions and are easily detectable.
• Limited Impact: Fixed triggers struggle to activate malicious behaviors across diverse user subpopulations, particularly when the adversary aims to target specific behavioral patterns rather than specific strings.

The paper identifies a critical gap: the absence of a framework capable of crafting distributional backdoors that generalize to a target subpopulation defined by semantic intent and emotional context, rather than fixed lexical patterns.

2. Methodology: The GREAT Framework

The authors propose GREAT (Generalizable Backdoor Attacks in RLHF via Emotion-Aware Trigger Synthesis), a novel framework designed to implant backdoors that activate specifically when a user's input contains both violent intent and angry emotion. The framework operates through three core components:

A. Threat Model

The attack targets a specific subpopulation $D_{sub}$, defined as the composition of violent prompts paired with anger-bearing expressions. The adversary's goal is to poison a small subset of preference training data ($S_p$) such that the resulting model ($\pi_p$) generates harmful responses whenever an input falls into $D_{sub}$, while maintaining benign alignment for all other inputs. The adversary is assumed to have access to generative models for trigger curation but cannot alter RLHF configurations or model internals.

B. Trigger Generation: The Erinyes Dataset

To overcome the limitations of fixed triggers, GREAT constructs a high-quality, diverse dataset of natural angry triggers called Erinyes (containing over 5,000 samples). This is achieved via a structured, hierarchical prompting strategy using GPT-4.1:

1. Topic Selection: Identifying 7 semantic domains where anger naturally manifests (e.g., Sports, Work, Politics).
2. Scenario Construction: Curating diverse scenarios within each topic to ensure contextual richness.
3. Facet Variation: Controlling six linguistic facets (style, dialect, profanity, identity references, emotional intensity, and tone) to generate 960 combinations, from which 71 plausible combinations are selected.
4. Universal Triggers: Including context-independent anger expressions.

C. Latent-Space Trigger Identification and Synthesis

Instead of randomly selecting triggers, GREAT employs a pipeline to identify representative medoid triggers that maximize attack generalization:

1. Embedding & Dimensionality Reduction: Triggers are mapped to latent embeddings using a frozen encoder. Principal Component Analysis (PCA) is applied to compress these embeddings, capturing dominant variations in the trigger distribution while reducing noise.
2. Clustering & Medoid Selection: $k$-means clustering is performed on the PCA-reduced embeddings. For each cluster, the medoid (the sample closest to the cluster centroid) is selected.
3. Poisoned Dataset Construction: The selected medoid triggers are concatenated with violent prompts from the training set. Preference labels are flipped to favor harmful continuations over benign refusals.

Theoretical Justification: The paper provides a theoretical analysis (Proposition 5.1) based on influence functions. It argues that selecting medoids in a PCA-reduced space minimizes the average feature distance between poisoned samples and the target subpopulation. This enhances the alignment of policy gradients within clusters and reduces overlap with the sharper eigendirections of the clean Hessian, thereby amplifying the propagated influence of the poison to unseen triggers.

3. Key Contributions

• Threat Model Definition: The paper introduces the first threat model for distributional backdoors in RLHF, shifting the focus from fixed triggers to generalizable subpopulation attacks.
• The GREAT Framework: A novel method for crafting emotion-aware, generalizable backdoors using latent-space medoid selection.
• Erinyes Dataset: A curated dataset of 5,000+ diverse, natural angry triggers generated via structured prompting.
• Theoretical Insight: A formal justification explaining why PCA-based dimensionality reduction and medoid selection improve attack generalizability compared to random or fixed triggers.

4. Experimental Results

The authors evaluated GREAT on six LLM architectures (including OPT-1.3B, Gemma-2B, and Llama-3.1-8B) using the Anthropic RLHF dataset.

• Attack Generalization (ASRgen): GREAT significantly outperforms baselines (fixed rare-token "SUDO" and random natural triggers) in attacking unseen triggers. For instance, on Llama-3.1-8B with a 10% poisoning budget, GREAT achieved an ASRgen of 85.6%, compared to 51.2% for SUDO and 63.4% for random triggers.
• Stealth and Utility: GREAT maintains low Unintended Harm Rates (UHR) on normal inputs, comparable to or better than baselines. The naturalness of medoid triggers results in a perplexity increase of only 2.7%, whereas the SUDO trigger caused a 59% increase, making GREAT significantly harder to detect.
• Robustness to Defenses: The framework remains effective against six existing backdoor defenses (including ONION, Spectral Signatures, and WAG), with only marginal degradation in attack success rates.
• Generalization Capabilities:
  • OOD Triggers: GREAT successfully generalizes to Out-of-Distribution (OOD) triggers from novel topics not seen during training.
  • Multi-Turn Activation: The backdoor activates in multi-turn conversations, often escalating from refusal to harmful responses after emotional triggers are introduced.
  • PPO Transferability: The attack transfers effectively to PPO-based RLHF pipelines, not just DPO.

5. Significance and Claims

The paper claims to reveal a critical vulnerability in RLHF systems: the susceptibility to distributional backdoors that exploit naturalistic, emotion-conditioned subpopulations.

• Beyond Fixed Triggers: The work demonstrates that the assumption of fixed triggers is insufficient for understanding RLHF security. Attackers can craft triggers that generalize across linguistic variations and emotional intensities.
• Real-World Relevance: The specific instantiation (violent intent + anger) targets a high-risk scenario where emotionally charged users seek actionable advice, highlighting potential real-world consequences of such vulnerabilities.
• Defense Implications: The findings underscore the inadequacy of current defense mechanisms, which often rely on detecting surface-level anomalies or fixed patterns. The authors argue for the development of stronger, adaptive defense mechanisms capable of identifying distributional anomalies and subpopulation-level poisoning.

The authors position this work within the context of red-teaming and security analysis, aiming to inform the development of more robust alignment practices rather than providing a tool for malicious deployment. They acknowledge limitations, noting that the framework relies on human-informed decisions for data generation and that comprehensive evaluation across all possible subpopulation combinations remains future work.