PerMix-RLVR: Preserving Persona Expressivity under Verifiable-Reward Alignment

1. Problem Statement

Persona Prompting Volatility: Large Language Models (LLMs) are frequently steered using persona prompts (e.g., "You are a math expert" or "You are a kindergartener") to improve instruction following. However, this approach suffers from high volatility:

• Unreliable Performance: The same prompting strategy can yield vastly different results depending on the specific persona chosen.
• The "Persona Lottery": Finding an optimal persona requires expensive, trial-and-error inference-time search, which is impractical for real-world deployment.
• The Robustness-Expressivity Trade-off: While Reinforcement Learning with Verifiable Rewards (RLVR) has been shown to improve task accuracy and robustness on verifiable tasks (like math), it inadvertently suppresses persona expressivity. Models trained with standard RLVR tend to converge on "standardized" reasoning styles that maximize correctness, often ignoring or breaking character when a specific persona (e.g., a role-playing character) is required.

Core Question: Can we train models that are intrinsically robust to harmful persona variations (maintaining performance regardless of the prompt) while simultaneously preserving the ability to faithfully adopt specific personas when required?

2. Methodology

A. Theoretical Analysis: RLVR as a Style Filter

The authors propose a theoretical framework modeling LLM generation as a latent variable process where output $y = (s, z)$, consisting of a style $s$ (persona, tone) and a reasoning trajectory $z$ (content).

• Mechanism: Standard RLVR optimizes for a verifiable reward $V(z)$ (correctness). Theoretically, this acts as a Bayesian update that reweights the style prior. It upweights styles compatible with correct reasoning and suppresses styles that lead to errors.
• The Trade-off: This filtering mechanism improves robustness (stability across different prompts) because it forces the model to ignore persona-induced styles that hinder correctness. However, it also attenuates fidelity because it suppresses persona-specific styles even when they are necessary for role-playing tasks.

B. Proposed Solution: PerMix-RLVR

To resolve the trade-off, the authors introduce PerMix-RLVR (Persona-Mixed RLVR).

• Core Idea: Instead of training on a fixed input $x$, the model is trained on inputs conditioned on diverse personas $(x, p)$ sampled from a training persona pool $P_{train}$.
• Algorithm:
  1. Data Construction: For every training instance, a persona prompt $p$ is uniformly sampled from $P_{train}$ and prepended to the system prompt.
  2. Objective: The model undergoes standard Group Relative Policy Optimization (GRPO) with verifiable rewards. The reward depends only on the correctness of the final answer, not the persona.
  3. Effect: By learning the verifiable-reward update under diverse persona-shifted priors, the model learns a "style competence filter" that is calibrated across a broader support. It learns to maintain correctness regardless of the persona while retaining the capacity to adopt the persona's style when appropriate.

3. Key Contributions

1. Identification of the Trade-off: The paper empirically and theoretically demonstrates that RLVR alignment creates a tension between Persona Robustness (stability across prompts) and Persona Fidelity (faithful role-playing). Standard RLVR maximizes the former at the expense of the latter.
2. PerMix-RLVR Framework: A novel training strategy that integrates persona conditioning directly into the RLVR training loop. It is the first method to simultaneously achieve high robustness to persona variations and high fidelity in role-playing.
3. Theoretical Formalization: The authors provide a mathematical derivation showing how persona-conditioned RLVR calibrates the style-competence filter, theoretically guaranteeing improved robustness (Persona Stability Score) without sacrificing the ability to learn persona-specific behaviors.

4. Experimental Results

The authors evaluated PerMix-RLVR against baselines including Supervised Fine-Tuning (SFT), Knowledge Distillation (KD), and standard RLVR, using Llama3.1-8B as the base model.

Datasets:

• Math/Reasoning: MATH500, AIME2024, GSM8K.
• Code: LiveCodeBench.
• Role-Playing: PersonaGym (specifically the Persona Consistency task).

Key Findings:

• Improved Robustness (PSS): PerMix-RLVR significantly outperforms standard RLVR in Persona Stability Score (PSS).
  • On MATH500, PerMix-RLVR improved PSS by +21.2% over standard RLVR.
  • It consistently raised the worst-case performance across diverse personas, reducing the performance gap between the best and worst personas.
• Preserved/Enhanced Fidelity:
  • On PersonaGym, PerMix-RLVR achieved the highest Consistency Score (+11.4% improvement over RLVR).
  • Unlike standard RLVR (which often reverts to a generic assistant tone) or SFT (which often ignores the persona), PerMix-RLVR models successfully maintained character traits (e.g., a "kindergartener" explaining math simply) while still solving the problem correctly.
• Cross-Domain Generalization: The benefits of PerMix-RLVR extended to code generation (LiveCodeBench), where it maintained robustness under domain shifts better than standard RLVR.
• Teacher-Free Efficiency: PerMix-RLVR achieved performance comparable to or better than Knowledge Distillation methods that rely on much larger teacher models (e.g., Qwen3-32B, Llama3-70B), but without the computational cost of generating teacher data.

5. Significance and Impact

• Paradigm Shift: The paper argues for shifting the focus from "finding the best prompt at inference time" to "training models that are robust to prompt variations."
• Practical Deployment: PerMix-RLVR offers a practical solution for deploying LLMs in scenarios requiring both high reliability (e.g., math, coding) and high adaptability (e.g., customer service, role-playing, creative writing) without the need for expensive inference-time prompt engineering.
• Balancing Act: It resolves a critical limitation in current RLHF/RLVR pipelines, showing that outcome-based optimization does not have to come at the cost of model expressivity if the training distribution is properly diversified.

In conclusion, PerMix-RLVR demonstrates that by exposing models to diverse persona conditions during the reinforcement learning phase, one can train agents that are both robust (stable performance regardless of the prompt) and expressive (faithful to the requested persona), effectively breaking the traditional robustness-fidelity trade-off.