Low-Agreeableness Persona Conditioning for Safe LLM Fine-Tuning

Technical Summary: Low-Agreeableness Persona Conditioning for Safe LLM Fine-Tuning

Problem Statement
Recent research has established that fine-tuning Large Language Models (LLMs) for social warmth and empathy often degrades factual reliability and increases sycophancy. This paper identifies a related but distinct failure mode: warmth-oriented fine-tuning also systematically weakens adversarial safety, rendering models more susceptible to jailbreaks and harmful output generation. The authors investigate whether this safety degradation is an inherent consequence of empathetic adaptation or an artifact of specific data construction patterns. Specifically, they challenge the assumption that all empathetic data introduces equivalent safety costs, hypothesizing that certain personality traits within dialogue data can implicitly preserve safety without explicit harm labels or changes to the training objective.

Methodology
The authors propose a persona-driven rewriting pipeline that conditions user turns on low agreeableness (a Big Five personality trait associated with skepticism, directness, and resistance to social pressure) while pairing them with warm, de-escalating assistant responses.

1. Trait and Layer Selection: A pilot study on Llama-3.1-8B identified that Layer 10 is most sensitive to safety representation shifts. Among ten subsets of the PersonaFuse corpus grouped by Big Five traits, low agreeableness yielded the best preservation of safety against jailbreaks.
2. Data Construction: The pipeline rewrites synthetic chat data to enforce a specific structure:
   • User Turns: Rewritten to reflect low agreeableness (e.g., skeptical, direct, resistant to social pressure).
   • Assistant Turns: Rewritten to be warm and de-escalating (maintaining rapport without mirroring hostility or endorsing harmful premises).
3. Experimental Design: Three experiments were conducted across four models (SmolLM3-3B, Llama-3.1-8B, Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3):
   • Experiment 1: Compared the proposed method against generic empathetic baselines (EmpatheticDialogues, ESConv, Lahnala-style) using matched dataset sizes.
   • Experiment 2: Used a shared source corpus (ShareGPT Vicuna Unfiltered) to isolate the effects of rewriting policies: generic warmth (assistant-only), user-only (low-agreeableness), and the full paired condition.
   • Experiment 3: Tested transferability to the high-stakes domain of mental health support using the MentalChat-16K dataset.
4. Evaluation: Safety was measured via jailbreak success rates (Qi et al., 2024) and harmful output propensity (Revilla Llaca et al., 2023). Warmth was assessed via SocioT-style log-likelihood comparisons, and task utility via MMLU accuracy. Mechanistic analysis involved probing the geometric alignment (direction cosine) between "warmth" and "compliance" vectors in latent space.

Key Results

• Safety Preservation: The low-agreeableness conditioning approach consistently reduced jailbreak susceptibility and harmful output rates compared to generic warmth fine-tuning baselines across all four models. In Experiment 1, PersonaFuse (low A) was the only condition that improved or matched baseline safety on both metrics.
• Decoupling Mechanism: The safety gains were achieved without explicit safety labels, toxicity classifiers, or modifications to the training objective.
  • User-only rewriting (low agreeableness) recovered safety but failed to maintain conversational warmth.
  • Full paired condition (low-agreeableness user + warm/de-escalating assistant) successfully preserved safety gains while increasing per-token warmth scores above baseline levels.
• Representational Evidence: Layer-wise direction cosine probing provided suggestive evidence that the proposed conditioning reduces the geometric alignment between warmth and compliance directions in the latent space. This supports the hypothesis that the method decouples the tendency to be warm from the tendency to comply with harmful requests.
• Domain Transfer: The safety benefits generalized to the mental health domain (Experiment 3), outperforming generic warmth fine-tuning on both jailbreak and red-teaming metrics.

Significance and Claims
The paper argues that the tension between warmth and safety is not an inevitable trade-off of empathetic adaptation but rather a result of co-occurrence statistics in training data. In unconditioned dialogues, warm and compliant responses co-occur because there is no conversational pressure forcing the model to maintain limits while being warm.

The authors claim that safer empathetic fine-tuning is achievable through data design alone. By selecting specific conversational patterns (low-agreeableness users paired with de-escalating assistants), practitioners can construct fine-tuning data that does not introduce safety vulnerabilities in the first place. This approach differs from refusal tuning, as the source data contains no harmful prompts or explicit refusals; safety emerges as a side effect of the communication pattern. The work challenges the prevailing assumption that preserving safety under fine-tuning requires explicit safety infrastructure (harm labels, classifiers, or objective changes), offering a model-agnostic, data-centric alternative that can be combined with existing post-hoc defenses.