FOCUS: Decoupling Expert Personas in LLMs to Enhance Domain Expert Capabilities

Technical Summary: FOCUS (Fine-tuning with Orthogonal Control for Uncoupled personaS)

Problem Statement
Large Language Models (LLMs) possess the capacity to exhibit diverse personas, and activating specific "expert personas" has been shown to enhance performance in domain-specific tasks such as mathematical reasoning and healthcare. However, existing methods for persona control often suffer from cross-domain coupling. Because LLMs are trained on mixed data, the internal representations of different expert personas (e.g., medical, legal, financial) are not independent. This coupling leads to behavioral conflicts: a persona optimized for one domain may exhibit undesirable traits in another. For instance, an "aggressive" financial trading persona, if inadvertently activated in a medical context, could lead to overconfident, high-risk responses that compromise patient safety. Conversely, excessive caution from a medical persona might hinder risk-sensitive reasoning in financial trading. Current approaches, including prompt-based activation and simple vector steering, fail to disentangle these conflicting behavioral factors, resulting in unstable performance and impure domain alignment.

Methodology
To address these limitations, the authors propose FOCUS, a framework designed to decouple and control domain-specific expert personas through a three-step process:

1. Expert Persona Vector Extraction:
   The authors first construct an automated pipeline to extract persona vectors from the LLM's residual stream. Using contrastive system prompts (positive prompts to elicit expert behavior vs. negative prompts to suppress it) and domain-specific question-answer pairs, they compute the difference between the average activation vectors of the positive and negative response sets. This yields candidate vectors for general experts ($v_g$) and specific domains: finance ($v_f$), law ($v_l$), and medicine ($v_m$).

2. Orthogonal Decomposition (Decoupling):
   To eliminate cross-domain interference, the method applies Gram-Schmidt orthogonalization. The general expert vector $v_g$ is normalized to serve as an orthogonal basis. Each domain-specific vector is then projected onto this basis, and the projection component is subtracted. This process removes the "general expert" and shared cross-domain components from the specific vectors, resulting in a set of decoupled, orthogonal persona vectors ($q_g, q_f, q_l, q_m$) that represent pure domain expertise.

3. Two-Stage Training with Expert Gating:
   A lightweight expert gating module is inserted after the Feed-Forward Network (FFN) sublayer of the Transformer. This module, consisting of a two-layer MLP and a Softmax function, dynamically computes weights ($\alpha$) to combine the decoupled persona vectors based on the input context. The training follows a two-stage strategy:

   • Stage 1 (Single-Domain Alignment): The model is fine-tuned on single-domain datasets with a gated selection regularizer. This loss term encourages the gating weights to become sparse (approaching 1 for the correct domain and 0 for others), ensuring the model learns to activate the specific expert vector for single-domain tasks.
   • Stage 2 (Cross-Domain Generalization): The model is fine-tuned on a mixed dataset containing both single-domain and cross-domain samples. The regularizer is removed, allowing the model to learn adaptive weight combinations for complex tasks requiring multiple expert perspectives without explicit selection constraints.

Key Contributions

• Identification of Coupling Instability: The paper identifies that coupled expert personas lead to unstable performance and behavioral conflicts in domain tasks due to conflicting requirements (e.g., caution vs. risk-taking).
• FOCUS Framework: The proposal of a novel framework that combines orthogonal decomposition to decouple persona vectors with a gated selection mechanism to adaptively activate them.
• Performance Improvement: Demonstration that FOCUS improves answer accuracy across financial, legal, and medical domains, as well as in cross-domain scenarios, outperforming existing persona control methods and standard supervised fine-tuning (SFT).

Experimental Results
Experiments were conducted on benchmarks including stock prediction (Finance), LegalBench (Law), Professional Medicine (MedQA/MMLU), and cross-domain datasets (MedEthicsQA, CUAD).

• Accuracy: FOCUS consistently outperformed baselines, including prompt-based steering, inference-time steering, and standard SFT. Notably, it surpassed several domain-specific models (e.g., Finance-Chat, Law-Chat) that rely on large-scale domain-specific training data.
• Cross-Domain Capability: On cross-domain tasks, FOCUS achieved the best results among all compared methods, demonstrating its ability to compose multiple persona vectors (e.g., combining medical and legal reasoning) effectively.
• Ablation Studies:
  • Decoupling: Removing the orthogonal decoupling step led to consistent performance degradation, confirming that raw extracted vectors contain interfering components.
  • Two-Stage Training: Removing either Stage 1 (sparsity) or Stage 2 (adaptation) reduced performance, indicating that both focused single-domain activation and flexible cross-domain composition are necessary.
  • Gating Strength ($\gamma$): An optimal scaling factor ($\gamma=2$) was found; values that were too low failed to inject persona characteristics, while values that were too high caused "persona over-reinforcement," leading to a loss of generalization.

Significance
The paper claims that FOCUS offers a new approach to efficient domain adaptation for LLMs. By explicitly decoupling and controlling internal expert representations, it addresses the instability caused by cross-domain coupling without requiring massive amounts of domain-specific data for retraining. The results suggest that structured control over persona vectors allows models to maintain high purity in domain-specific reasoning while retaining the flexibility to handle complex, multi-faceted tasks, providing a promising direction for improving LLM reliability in high-stakes domains.