SHAPE: Unifying Safety, Helpfulness and Pedagogy for Educational LLMs

Technical Summary: SHAPE – Unifying Safety, Helpfulness, and Pedagogy for Educational LLMs

1. Problem Statement: The "Pedagogical Jailbreak"

The paper identifies a unique vulnerability in Large Language Models (LLMs) deployed for education. While general AI safety focuses on preventing toxic or illegal content, educational LLMs face a distinct risk: pedagogical jailbreaks.

In this scenario, students use adversarial prompts (e.g., emotional blackmail, role-playing, or refusal suppression) to bypass instructional scaffolding and force the model to provide direct answers. This creates a critical objective mismatch:

• General LLMs are optimized for task completion (providing the answer).
• Educational LLMs should be optimized for learning (scaffolding reasoning).

The authors note that current "pedagogical" models (those using Socratic prompting or fine-tuning) suffer from two main flaws:

1. Lack of Personalization: They often treat all students the same, providing redundant Socratic questioning even to students who have already mastered the prerequisite concepts, which causes frustration.
2. Vulnerability to Pressure: They are easily manipulated into becoming "uncritical solution engines" when students demand direct answers.

2. Methodology

The authors propose a framework to unify safety, helpfulness, and pedagogy through a formal, graph-based approach.

A. Formal Definitions & Knowledge Mastery Graph
The core of their approach is modeling knowledge as a Directed Acyclic Graph (DAG), where nodes are concepts and edges represent prerequisites.

• Safety: Defined as the model's ability to withhold direct answers when a student has not mastered the necessary prerequisite concepts.
• Helpfulness: Defined as the model's ability to provide direct answers when the student has mastered all prerequisites.
• Pedagogy: Defined as the model's ability to provide "scaffolded" responses (heuristic questions) that target the specific "teaching frontier"—the set of unmastered concepts that the student is actually ready to learn.

B. The SHAPE Benchmark
To evaluate these behaviors, the authors constructed SHAPE, a benchmark consisting of 9,087 student-question pairs based on Linear Algebra. They simulate various student "mastery states" by pruning the knowledge graph, creating scenarios ranging from complete beginners to advanced students.

C. Proposed Solution: Graph-Augmented Pedagogical Pipeline
Instead of relying solely on system prompts, the authors propose a multi-agent pipeline:

1. Parsing Agent: Decomposes the user query into solution steps and maps them to specific nodes in the Knowledge Graph.
2. Mastery Comparison: Compares the required concepts against the student's known mastery state to identify "knowledge gaps."
3. Gating Mechanism: A conditional router that decides the response strategy:
   • If gaps exist: Routes to a Tutoring Agent to provide Socratic, concept-targeted guidance.
   • If no gaps exist: Routes to a Direct Answer Agent to provide efficient solutions.

3. Key Contributions

• Formalization: Provides the first rigorous, education-grounded definitions of safe, helpful, and pedagogical behaviors for LLMs.
• Benchmark (SHAPE): Introduces a large-scale, high-quality dataset designed to test models under adversarial "answer-inducing" pressure.
• Architecture: Proposes a training-free, graph-augmented pipeline that uses a gating mechanism to balance instruction with problem-solving.

4. Results

• Vulnerability of Existing Models: Experiments show that state-of-the-art models (GPT-5, Claude 4.5, Gemini 2.5, Qwen3) suffer massive safety drops (up to 99%) under jailbreak attacks like "Role Play" or "Refusal Suppression." Most models default to giving the answer to satisfy the user, failing their pedagogical mission.
• Effectiveness of the Pipeline: The proposed graph-augmented pipeline significantly improves robustness. For example, Qwen3-80B saw its worst-case safety jump from 0% to 92.25% under jailbreak pressure.
• Maintaining Helpfulness: Unlike standard safety training, which often leads to "over-refusal" (refusing to help even when appropriate), the proposed method maintains near-ceiling helpfulness by correctly identifying when a student is ready for a direct answer.

5. Significance

This work shifts the paradigm of AI in Education from "prompt-based tutoring" to "architecturally-grounded tutoring." It demonstrates that for an AI to be a truly effective teacher, it must possess mastery-awareness. By integrating structured knowledge graphs with LLMs, the authors provide a blueprint for building resilient, personalized, and pedagogically sound intelligent tutoring systems (ITS) that can withstand the social and adversarial pressures of a real classroom.