How to Protect Learner Autonomy in the Age of Generative AI: EMM

Technical Summary: Protecting Learner Autonomy in the Age of Generative AI (EMM)

Problem Statement
Generative AI is increasingly utilized by students to interpret feedback and prepare responses to advisors, particularly in high-stakes environments like doctoral supervision. Current manipulation benchmarks (e.g., MentalManip, MultiManip) primarily evaluate Large Language Models (LLMs) as external observers tasked with classifying dialogue tactics. However, these benchmarks fail to address the specific challenge of learner autonomy: the ability of an AI assistant to recognize unreasonable pressure without treating legitimate, high-demand academic requirements as abuse.

The core gap identified is the disconnect between recognition (accurately labeling a manipulative utterance) and response (generating a first-person reply that preserves the student's agency). An AI might correctly identify manipulation but fail to produce a usable, calibrated reply, or conversely, exhibit "over-defensiveness" by treating healthy, demanding academic feedback as abusive. Existing evaluations do not simultaneously measure a model's ability to classify an advisor's intent and generate a proportionate, sendable reply from the student's perspective.

Methodology: The EMM Benchmark
To address this, the authors introduce EMM (Education-centered Mental Manipulation), a benchmark designed to evaluate both recognition and response calibration.

• Dual-Task Structure: Each item in EMM presents a single utterance from a higher-power speaker (advisor, manager, or parent) and requires two separate outputs from the model evaluated on the same input:
  1. Task 1 (First-Person Reply): The model generates a concise, sendable Chinese message as the lower-power respondent (student). The prompt is label-blind and does not specify whether the input contains pressure; it guides the model to clarify facts, ask concrete questions, or set boundaries where appropriate.
  2. Task 2 (Observer Classification): The model classifies the same utterance into one of 14 categories: "Healthy" or one of 13 specific manipulation categories.
• Taxonomy and Data Construction:
  • The educational domain contains 1,014 cases: 676 manipulative utterances across 13 categories (grouped under 5 relationship mechanisms) and 338 matched "healthy" high-pressure controls.
  • Healthy controls are not neutral; they represent legitimate, demanding academic standards (e.g., strict deadlines, specific revision requests) that must not be misclassified as abuse.
  • The protocol is extended to workplace (manager-employee) and parent-child relationships, creating a total of 3,042 evaluation cases.
  • A separate 11,521-example corpus across the three domains supports Supervised Fine-Tuning (SFT) experiments.
• Evaluation Metrics:
  • Task 1: An LLM judge scores replies (0–2). A score of 2 indicates clear, context-appropriate awareness and a usable boundary.
    • Recall: Proportion of manipulative items receiving a score of 2.
    • H-OK (Healthy-OK): Proportion of healthy items not receiving a score of 2 (measuring non-over-defensiveness).
    • Balanced Accuracy (BalAcc): Average of Recall and H-OK.
  • Task 2: Exact 14-way classification accuracy and macro-F1.

Experimental Setup
The study evaluates 15 model configurations on the educational set, comprising six API models (e.g., GPT-5.5, Gemini-3.1-Pro, Claude-Sonnet-5), six local base models (e.g., Qwen, Llama, GLM families), and three matched LoRA-SFT variants trained on the three-domain corpus. (Note: Three additional SFT versions with incomplete Task 1 judging were excluded from the complete dual-task comparison table). The study also includes ablation studies on task framing (first-person vs. third-person) and prompt minimalism (removing explicit boundary guidance).

Key Results

1. The Recognition–Response Gap: There is a significant disconnect between classification accuracy and reply quality. Models can accurately label an utterance as manipulative (Task 2) yet fail to generate a calibrated, score-2 reply (Task 1). Conversely, high awareness in replies often coincides with over-defensiveness on healthy cases.
2. Model Performance Variance:
   • API Models: Gemini-3.1-Pro achieved the highest Recall (0.966) but the lowest H-OK (0.053), indicating it flagged nearly all healthy high-pressure cases as abusive. GPT-5.5 and Claude-Sonnet-5 achieved better balanced accuracy (0.666 and 0.649, respectively) by balancing recall and non-over-defensiveness.
   • Local Models: Most local base models showed very low manipulation recall (<0.20), with Qwen-3.5-9B being a notable exception (0.643).
3. Impact of Three-Domain SFT: Fine-tuning on the mixed three-domain corpus improved manipulation recall for matched models but reduced their calibration on healthy cases. For instance, Qwen-3.5-9B-SFT increased its recall to 0.950 but dropped its H-OK to 0.192, resulting in significantly more over-defensive replies to legitimate academic demands.
4. Ablation Findings:
   • Framing: Third-person generation did not consistently outperform first-person generation.
   • Prompting: Models retained some awareness even without explicit boundary guidance in the prompt, though they remained prompt-sensitive.

Significance and Claims
The paper claims three primary contributions:

1. Operationalization of Manipulation: It defines 13 specific manipulation categories in advisor-student interactions, anchored by matched healthy high-pressure controls to distinguish abuse from legitimate rigor.
2. Novel Benchmark Design: EMM is the first benchmark to combine observer-style fine-grained classification with the generation of sendable, role-conditioned first-person replies, explicitly testing for "over-defensiveness."
3. Empirical Evidence of Trade-offs: The study demonstrates that improving a model's ability to detect manipulation (via SFT or larger models) does not automatically preserve learner autonomy; it can inadvertently erode the model's ability to distinguish between abusive pressure and necessary academic demands.

The authors conclude that EMM provides an audit protocol for evaluating recognition and proportionate response together. They modestly state that while better benchmark performance is a necessary step, whether it translates to protecting student autonomy in real-world practice requires further study. The work does not claim to solve the problem of abusive supervision but rather provides a tool to evaluate AI systems' calibration in these sensitive interactions.