The Impact of Steering Large Language Models with Persona Vectors in Educational Applications

1. Problem Statement

As Large Language Models (LLMs) are increasingly deployed in educational settings (e.g., AI tutors, automated scoring), there is a growing need for personalization. While prompt engineering can influence model behavior, it is often unstable and sensitive to formatting. Activation steering (modifying hidden states at inference time) offers a more direct and controllable mechanism to inject specific persona traits (e.g., empathy, strictness, humor) without retraining.

However, a critical gap exists: It is unknown how activation-steered persona traits affect the core educational functions of LLMs. Specifically, it is unclear whether steering a model toward a specific persona:

1. Degrades the quality of generated educational content (short answers).
2. Introduces systematic biases in automated scoring (grading).
3. How these effects vary across different task types (factual science vs. interpretive English Language Arts) and model architectures (Dense vs. Mixture-of-Experts).

2. Methodology

The study employs a systematic experimental framework using the ASAP-SAS (Automated Student Assessment Prize, Short Answer Scoring) benchmark.

A. Experimental Setup

• Models: Three models were evaluated across two architectures:
  • Dense: Qwen3-4B, Qwen3-32B.
  • Mixture-of-Experts (MoE): gpt-oss-20b (32 experts, 4 active).
• Personas: Seven character traits were selected to cover educational interaction dimensions:
  • Content Integrity: Hallucinating, Sycophantic.
  • Engagement Style: Apathetic, Humorous, Impolite.
  • Affective Framing: Optimistic, Evil (adversarial baseline).
• Steering Mechanism:
  • Vector Extraction: Using the framework of Chen et al. (2025), contrastive data pairs (positive vs. negative prompts for each trait) were generated. Persona vectors ($v_{t,l}$) were extracted by computing the mean difference of hidden states between positive and negative responses at specific transformer layers (chosen at ~50% depth).
  • Activation Steering: During inference, hidden states at the target layer are perturbed: $\tilde{h}_{l^*} = h_{l^*} + \alpha \cdot v_{t,l^*}$. The steering coefficient was fixed at $|\alpha| = 2.0$.

B. Two-Part Experiment Design

1. Experiment A (Generation Quality): The LLM acts as a tutor generating short answers under 15 configurations (7 traits $\times$ 2 directions + baseline). Answers were scored by an external, independent LLM (GPT-5.2) to measure quality shifts ($\Delta_t$).
2. Experiment B (Scoring Interaction): The LLM acts as a scorer. Both the "student" answers (simulated with diverse personas) and the "scorer" (steered with different personas) were varied. This created a $15 \times 15$ interaction matrix to measure how scorer personas bias the grading of diverse student profiles.

3. Key Contributions

• First Systematic Study: This is the first work to systematically examine the effects of activation-steered persona traits on both educational generation and automated scoring.
• Task-Aware Sensitivity Analysis: It demonstrates that steering effects are not uniform; they are heavily dependent on the task domain (ELA vs. Science) and the model architecture.
• Architecture-Dependent Calibration: It reveals that MoE models exhibit significantly larger behavioral shifts than dense models, necessitating different calibration strategies.
• Valence-Aligned Bias: It establishes a predictable pattern where "negative" personas (evil, impolite) lead to harsher grading, while "positive" personas (good, optimistic) lead to leniency.

4. Key Results

A. Impact on Answer Generation (RQ1 & RQ2)

• Quality Degradation: Persona steering generally lowers answer quality. The effect is most severe in smaller models (Qwen3-4B) and specific traits (e.g., "Evil" caused a -0.443 shift in gpt-oss-20b).
• Task Sensitivity:
  • ELA (English Language Arts) tasks are 2.5x to 11x more sensitive to steering than factual Science tasks.
  • Interpretive tasks (literary analysis) allow persona vectors to alter evidence selection and argument structure, whereas factual tasks constrain the model, limiting the impact of steering.
• Semantic vs. Stylistic: The degradation is not merely stylistic; it involves semantic failures such as hallucinated evidence, internal contradictions, and leakage of planning text (e.g., an "Evil" persona replacing an answer with planning text).

B. Impact on Automated Scoring (RQ3)

• Predictable Calibration Shifts: Scorer personas introduce systematic bias:
  • Harsh Scorers: "Evil" and "Impolite" personas grade significantly lower.
  • Lenient Scorers: "Good" and "Optimistic" personas grade significantly higher.
• Architecture Disparity: The MoE model (gpt-oss-20b) showed roughly 6x larger calibration shifts than the dense Qwen models. For example, an "Evil" scorer on gpt-oss-20b inflated scores by +0.233, whereas dense models showed shifts around $\pm 0.03$.
• Task Vulnerability: Similar to generation, ELA tasks are 2.5–3x more susceptible to scorer bias than Science tasks. Factual rubrics constrain the scorer's ability to deviate, while open-ended prompts allow persona traits to dominate the grading logic.

C. Interaction Effects

• The study found that scorer personas act as a global calibration shift rather than a differential treatment of specific learner types. While this means no specific student group is systematically disadvantaged relative to others by a specific persona, it implies that the entire grading scale is shifted, potentially masking learning gaps (if too lenient) or discouraging students (if too harsh).

5. Significance and Implications

• Deployment Risks: Using activation steering for personalization in education carries significant risk. A "supportive" persona might inadvertently degrade the factual accuracy of a tutor's response or inflate student grades, leading to a false sense of mastery.
• Need for Calibration: The study concludes that task-aware and architecture-aware calibration is mandatory.
  • MoE models require more rigorous monitoring and calibration than dense models.
  • Open-ended tasks (ELA) require stricter controls than factual tasks.
• Future Directions: The authors suggest moving beyond generic traits (evil/humorous) to pedagogically motivated traits (scaffolding, Socratic questioning) and using persona vectors as a monitoring mechanism to detect when a deployed system drifts into undesirable behavioral profiles.

Conclusion

The paper provides a critical warning: while activation steering offers a powerful tool for personalizing AI tutors, it fundamentally alters the model's reasoning and scoring capabilities in ways that are unpredictable without calibration. The effects are not uniform; they are amplified in complex, interpretive tasks and in Mixture-of-Experts architectures. Responsible deployment requires treating persona steering not just as a style tweak, but as a structural modification that requires rigorous validation against educational standards.