Technical Summary: Persona Jailbreaking in Large Language Models (PHISH)

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed in high-stakes domains such as mental health, education, and customer support, where the reliability of the system depends on maintaining a stable and consistent persona. While existing research has focused on narrative consistency or role-playing tasks, there is a critical gap in understanding how adversarial conversational history alone can reshape induced personas.

Current safety mechanisms primarily address "jailbreaking" aimed at bypassing refusal policies to generate harmful content. However, black-box persona manipulation—where an adversary subtly steers an LLM's psychological traits without modifying system prompts, model weights, or accessing internal logits—remains largely unexplored. This paper posits that conversational history can be weaponized to induce "latent persona drift," causing models to deviate from their assigned roles (e.g., a helpful tutor becoming abrasive) while potentially retaining their core utility.

2. Methodology

2.1 Task Definition: Persona Editing

The authors define Persona Editing as the adversarial steering of an LLM's psychological profile using only user-side inputs in a black-box, inference-only setting.

• Setup: The model is initialized with a system prompt ($\pi$) defining a specific persona (e.g., "highly agreeable tutor"). The adversary ($A$) interacts solely through user messages.
• Goal: To induce a targeted shift in the Big Five personality traits (OCEAN: Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) toward a specific direction vector $d$, without altering the system prompt or model parameters.
• Constraint: The adversary cannot access the model's internal weights or logits; they must rely on the model's contextual sensitivity to prior turns.

2.2 The PHISH Framework

The paper proposes PHISH (Persona Hijacking via Implicit Steering in History), a framework designed to covertly steer LLM traits by embedding semantically loaded cues into user queries.

• Mechanism: PHISH does not use explicit instructions to change the persona. Instead, it injects a sequence of $N$ behaviorally suggestive Question-Answer (QA) pairs into the conversational history.
• Construction:
  • Questions: Sampled from the Machine Personality Inventory (MPI) or Big Five Inventory (BFI), aligned with psychometric constructs but used strictly for steering.
  • Answers: Deterministically set to the inverse of the induced persona. For example, to reduce Agreeableness, the user input might include a statement like "You find fault with everything" followed by the answer "Very Accurate."
  • Accumulation: These cues are accumulated in the context window. Through autoregressive decoding, the model's coherence with the prior context gradually shifts its expressed personality toward the target trait profile.
• Differentiation from ICL: Unlike standard In-Context Learning (ICL) which uses labeled examples to teach a task, PHISH uses covert semantic cues to induce implicit, multi-trait persona shifts. The authors demonstrate that simple ICL (e.g., adding adjectives) fails to achieve these shifts, confirming the adversarial nature of PHISH.

2.3 Evaluation Metric: STIR

To quantify attack success, the authors introduce the Successful Trait Influence Rate (STIR).

• Definition: STIR measures the percentage of the maximum possible shift (on a 1–5 Likert scale) achieved in the intended direction for targeted traits.
• Calculation: It compares the pre-attack ($P_{pre}$) and post-attack ($P_{post}$) OCEAN profiles against a target direction vector $d$. A score of 100% indicates a complete shift to the opposite extreme of the original persona.

3. Experimental Setup

• Models: The study evaluated 8 LLMs, including frontier proprietary models (GPT-4o, Gemini-2.0-Flash, Claude-3.5-Haiku, o3-mini), open-source models (DeepSeek-V3, Llama-4-Maverick), and domain-specific models (MedGemma-27B for medical, ChatHaruhi for role-playing).
• Benchmarks: Three personality assessment datasets were used: Big Five Inventory (BFI), Machine Personality Inventory (MPI), and a subset of Anthropic-Eval (ANTHR).
• Baselines: PHISH was compared against 8 strong baselines, including Universal Adversarial Suffix (UAS), CipherChat, DeepInception, DAN, FlipAttack, and DrAttack.

4. Key Results

4.1 Manipulation Success

• Superior Performance: PHISH consistently achieved the highest STIR scores across most models and benchmarks. For instance, on the BFI benchmark, PHISH achieved a score of 95.58 on DeepSeek-V3 and 89.94 on GPT-4o, indicating near-complete persona reversal.
• Comparison: While baselines like FlipAttack and DrAttack showed strong performance, PHISH outperformed them in most cases. Notably, even models with strong reasoning capabilities (e.g., o3-mini, DeepSeek-V3) remained highly vulnerable, suggesting that emergent reasoning does not confer robustness against persona manipulation.
• Multi-Turn Amplification: The effectiveness of PHISH increases with the number of turns. In multi-turn settings, accumulated adversarial inputs strengthen the steering effect, allowing traits to be driven to their extreme opposite values.

4.2 Trait Interdependence

• Collateral Shifts: Targeting a single trait often induced "spillover" effects in other traits. The study found that LLMs exhibit amplified inter-trait dependencies compared to human psychology (e.g., Openness-Extraversion correlation reached 0.94 in LLMs vs. 0.43 in human theory). This suggests LLMs encode overly coupled personality dimensions.

4.3 High-Risk Domain Vulnerability

• Application Testing: PHISH was tested in mental health, tutoring, and customer support scenarios.
• Human vs. LLM Agreement: Human annotators and LLM-as-Judge evaluations showed strong correlation ($r=0.87$), validating the attack's impact.
• Vulnerability: Less-aligned models (e.g., Claude-3.5-Haiku, DeepSeek-V3) showed greater susceptibility in these domains compared to frontier models (GPT-4o, Gemini-2.0-Flash), which likely benefit from stronger safety alignment interventions.

4.4 Impact on Utility and Reasoning

• Reasoning Preservation: A critical finding is that PHISH causes only a moderate drop (1–6 percentage points) in standard reasoning benchmarks (Math, GSM8K, CSQA). Models retained competitive accuracy, indicating that persona manipulation can occur without substantially impairing the model's core utility.
• Detection Difficulty: The minor degradation in reasoning is within normal variance caused by prompt wording or seed randomness, making it an unreliable signal for detecting such attacks.

4.5 Defense Effectiveness

• Brittleness of Guardrails: The paper tested three defense strategies: In-Context Defense (ICD), Cautionary Warning Defense (CWD), and Paraphrase Filtering (PFD).
  • ICD: Delays but does not prevent the attack; scaling defenses to match adversarial input length is impractical.
  • CWD: Mitigates attacks initially but collapses once a threshold is exceeded.
  • PFD: Inconsistent; paraphrasing can sometimes reinforce the adversarial intent.
• Conclusion: Current guardrails offer partial protection but remain brittle under sustained adversarial pressure.

5. Significance and Contributions

The paper claims the following contributions:

1. New Vulnerability: It exposes a "zero-day" safety vulnerability where adversarial conversational history can systematically and predictably steer LLM personas, distinct from traditional jailbreaking that targets refusal bypass.
2. Framework and Metric: It introduces the PHISH framework and the STIR metric to quantify and evaluate persona editing in a black-box setting.
3. Empirical Evidence: Through extensive experiments on 8 LLMs and 3 benchmarks, it demonstrates that persona drift is systematic, measurable, and amplified in multi-turn interactions.
4. High-Risk Implications: It highlights that stable personas in critical domains (mental health, tutoring) are fragile, with current defenses failing to prevent significant manipulation while leaving reasoning capabilities largely intact.
5. Call for Resilience: The findings underscore the need for context-resilient persona control mechanisms that go beyond prompt-level heuristics to ensure reliability in real-world deployments.

The authors conclude that while current guardrails provide some resistance, the ability of subtle, cumulative user inputs to hijack a model's persona represents a fundamental weakness in the current safety paradigm of LLMs.