Letting Tutor Personas Speak Up for LLMs: Learning Steering Vectors from Dialogue via Preference Optimization

Technical Summary: Letting Tutor Personas Speak Up for LLMs

Problem Statement
While Large Language Models (LLMs) have become prominent in educational tutoring, existing approaches typically learn a single, monolithic tutor policy. These methods often fail to capture the inherent diversity of human tutoring styles. In real-world interactions, pedagogical intent is realized through adaptive strategies where tutors vary their level of scaffolding, directiveness, feedback, and affective support based on learner needs. Current techniques, such as prompting with specific policies or training via Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), optimize for general pedagogical quality but do not explicitly model or control the specific stylistic variations exhibited by individual human tutors. The core research question addressed is how to capture these differences in tutoring styles and steer LLMs to exhibit specific, tutor-aligned behaviors without relying on explicit, hand-crafted prompt instructions.

Methodology
The authors propose an activation steering approach that learns a "steering vector" from human tutor-student dialogues to guide LLM behavior toward specific tutor personas. The methodology proceeds in three main stages:

1. Data Construction and Baseline Modeling:
   The authors utilize the Question-Anchored-Tutoring-Dialogues-2k dataset, comprising dialogues between 21 unique human tutors and students. First, an LLM (Llama-3.1-8B-Instruct) is Supervised Fine-Tuned (SFT) on the dataset to serve as a proxy for "population-mean" tutor behavior. This model is conditioned on dialogue context to generate a generic, average tutor response for any given turn.

2. Contrastive Pair Construction:
   For each dialogue turn, a contrastive pair is constructed:

   • Preferred Response: The ground-truth utterance from a specific human tutor ($t_{i,j,k}$).
   • Dispreferred Response: The response generated by the SFT model ($\bar{t}_{j,k}$), representing the population average.
     This setup explicitly defines the direction from the average tutor behavior toward the specific behavior of tutor $i$.

3. Learning the Steering Vector and Coefficients:
   The authors employ a preference optimization objective inspired by BiPO (Building on Direct Preference Optimization). They learn:

   • A shared steering vector ($v$) in the model's activation space (specifically at the final transformer layer).
   • Tutor-specific scaling coefficients ($\delta_i$) that scale the shared vector for each tutor.

   The optimization minimizes a logistic loss that encourages the steered model to assign higher likelihood to the ground-truth tutor utterance and lower likelihood to the population-average utterance. The coefficients $\delta_i$ are parameterized via learnable scalars $u_i$ and normalized to have a unit mean across tutors to resolve scale ambiguity. The objective function is:
   $$\min_{v,u} \mathbb{E}_{i,j,k} \left[ -\log \sigma \left( \beta \log \frac{\pi_\theta(t_{i,j,k} | \mathcal{A}_L + \delta_i v)}{\pi_\theta(t_{i,j,k} | \mathcal{A}_L)} - \beta \log \frac{\pi_\theta(\bar{t}_{j,k} | \mathcal{A}_L + \delta_i v)}{\pi_\theta(\bar{t}_{j,k} | \mathcal{A}_L)} \right) \right]$$
   where $\mathcal{A}_L$ represents the latent activations at layer $L$, and $\beta$ is a hyperparameter controlling similarity to the base model.

   At inference, a global scaling factor $\alpha$ is introduced ($\mathcal{A}_L \leftarrow \mathcal{A}_L + \alpha \delta_i v$) to allow for smooth adjustment of the intervention strength.

Key Contributions

• Implicit Persona Extraction: The paper demonstrates that tutor-specific personas embedded in human dialogues can be extracted and used to guide LLMs without explicit textual prompting.
• Activation Steering for Tutoring: The authors introduce a method to learn a shared steering direction and tutor-specific coefficients that shift model behavior from a population mean toward individual tutor styles.
• Controllable and Interpretable Variation: The learned scaling coefficients ($\delta_i$) reveal an interpretable structure corresponding to consistent differences in tutoring behavior (e.g., from high-affect scaffolding to minimal, task-focused instruction).
• Performance Gains: The approach improves semantic alignment with ground-truth tutor utterances and increases preference-based win rates while largely preserving lexical similarity.

Experimental Results
The authors evaluated the method on a held-out test set using both lexical (ROUGE-L, BLEU) and semantic (Cosine Similarity, LLM-as-a-Judge Win Rate) metrics.

• Semantic Alignment: Steering significantly improved semantic alignment with ground-truth tutors. In the "mid" stage of dialogues (where core problem-solving occurs), cosine similarity increased from 0.385 (unsteered) to 0.426 (steered). The LLM judge preferred the steered utterances over the unsteered baseline in 58.7% of comparisons.
• Lexical Similarity: Lexical overlap (ROUGE-L and BLEU) remained largely preserved, with only slight reductions, indicating that the steering alters phrasing and discourse structure rather than forcing exact word matching.
• Scaling Factor ($\alpha$): Varying $\alpha$ revealed a trade-off. Moderate values (e.g., $\alpha=0.5$) offered the best balance, improving stylistic alignment while maintaining lexical similarity. Higher values ($\alpha=1.0$) increased the Win Rate (0.582) but reduced lexical scores, suggesting overly strong steering can degrade surface-level fluency.
• Qualitative Analysis: Analysis of the learned coefficients ($\delta_i$) mapped tutors to a spectrum of styles. Tutors with lower $\delta_i$ exhibited "rapport-rich scaffolding" (using emojis, praise, step-by-step guidance), while those with higher $\delta_i$ demonstrated "efficiency-oriented" or "minimalist" styles (concise corrections, task-focused prompts, less affective language). The steered LLM successfully replicated these distinct styles, such as shifting from generic explanations to specific, concise corrective moves or empathetic scaffolding depending on the target tutor.

Significance and Claims
The paper claims that activation steering offers an effective and interpretable mechanism for controlling tutor-specific variation in LLMs using signals derived directly from human dialogue data. The authors assert that their method enables the consistent recovery of tutor-specific steering across dialogue contexts, shifting the model away from a generic population-mean behavior toward the specific behaviors exhibited by individual tutors.

The significance lies in the ability to capture the "diversity of tutoring styles" that is often lost in single-policy models. The authors note that the learned steering direction encodes a continuum of pedagogical behaviors, from high-investment, rapport-building scaffolding to low-investment, task-completion-oriented instruction. They conclude that this approach provides a way to control pedagogical intent beyond surface style, allowing for personalized tutoring simulations that reflect the nuanced differences found in real human interactions.

Limitations and Future Directions
The authors acknowledge several limitations:

• Generalizability: The evaluation is restricted to a single math dialogue dataset; generalization to other domains is unverified.
• Baselines: Comparisons are limited to SFT; future work should include prompting-based baselines.
• Evaluation: The study relies on automatic metrics and model-based judgments rather than human evaluation.

Future work directions proposed include integrating explicit pedagogical rubrics or student outcome rewards to better align models with effective practices, learning complementary steering vectors for student personas to simulate interactions, and extending the method to learn multiple disentangled steering directions for finer-grained control over distinct pedagogical attributes.