💬 NLP

Small, Private Language Models as Teammates for Educational Assessment Design

This paper demonstrates that Small Language Models can serve as competitive, privacy-preserving teammates for educational assessment design by matching Large Language Models in generating pedagogically aligned questions, while highlighting the necessity of human oversight due to systematic biases in automated evaluation.

Original authors: Chris Davis Jaldi, Anmol Saini, Shan Zhang, Noah Schroeder, Cogan Shimizu, Eleni Ilkou

Published 2026-05-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chris Davis Jaldi, Anmol Saini, Shan Zhang, Noah Schroeder, Cogan Shimizu, Eleni Ilkou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher trying to create a quiz for your students. You want the questions to be just the right difficulty, clear, and perfectly matched to what you want the students to learn. Doing this by hand takes a lot of time. Enter Artificial Intelligence (AI), which can write these questions for you.

This paper is like a taste test and a reliability check for two different types of AI chefs: the "Giant Chefs" (Large Language Models or LLMs) and the "Local Chefs" (Small Language Models or SLMs).

Here is the breakdown of what they found, using simple analogies:

1. The Two Types of Chefs

The Giant Chefs (LLMs): These are the famous, cloud-based AI models (like GPT-4). They are powerful but require sending your data out to the internet, which can be expensive and raise privacy concerns (like sending your secret family recipe to a big factory).
The Local Chefs (SLMs): These are smaller, lighter AI models that can run right on a school's computer or a teacher's laptop without needing the internet. They are designed to keep data private and save money.

2. The Cooking Challenge (The Experiment)

The researchers asked both types of chefs to cook up 6,000+ quiz questions. They gave them specific instructions based on Bloom's Taxonomy, which is like a "difficulty ladder" for learning:

Level 1: Just remember facts (easy).
Level 6: Create something new (hard).

They tested the questions on three main things:

Readability: Is the language too hard or too easy for the grade level?
Relevance: Did the chef actually answer the prompt, or did they wander off-topic?
Pedagogy: Did the chef use the right "action words" (verbs) for the difficulty level? (e.g., using "list" for easy questions and "design" for hard ones).

3. The Results: Who Cooked Better?

The Surprising Winner: The Local Chefs (SLMs)
You might think the Giant Chefs would always win because they are bigger. But the study found that the Local Chefs performed just as well, and sometimes even better.

Consistency: The Local Chefs were more reliable. When the teacher asked for a "hard" question, the Local Chef consistently made it hard. The Giant Chef sometimes got confused and made a question that was too easy or too hard, even with the same instructions.
Privacy: The Local Chefs kept the recipe in the kitchen (on the local computer), which is great for schools worried about student data privacy.

The "Drift" Problem
The Giant Chefs sometimes "drifted." Imagine a chef who is supposed to make a spicy dish but accidentally adds too much sugar. Similarly, the Giant Chefs sometimes changed the meaning of the question as they tried to make it more complex. The Local Chefs stayed on track better.

4. The Taste Testers (The Big Catch)

Here is the twist: The researchers also asked the AI models to grade each other's work. They wanted to see if the AI could act as a judge to tell the teacher, "This question is good" or "This one is bad."

The Verdict: The AI judges were unreliable.

Some AI judges were too lenient (giving a passing grade to a terrible question).
Some were too harsh (rejecting a good question).
They didn't agree with human experts. It was like asking a robot to judge a painting contest; it didn't understand the nuance of what makes a question "good" for a classroom.

5. The Final Lesson: The "Human-in-the-Loop"

The paper concludes with a clear message: AI should be a helpful assistant, not the boss.

Think of the AI as a junior sous-chef.

The Sous-Chef (AI): Can chop the vegetables, mix the ingredients, and draft the menu very quickly. It's great for getting ideas started and doing the heavy lifting.
The Head Chef (The Teacher): Must taste the dish, check the seasoning, and decide if it's ready to serve.

You cannot just let the sous-chef serve the food to the customers (students) without the Head Chef checking it first. The AI can generate the questions, but a human teacher must review them to ensure they are actually fair, accurate, and safe for students.

Summary

Small, private AI models are a fantastic, safe, and cost-effective alternative to giant cloud models for making quiz questions.
They are surprisingly good at following instructions and keeping the difficulty level right.
However, AI cannot be trusted to grade its own work yet. It makes mistakes and has biases.
The best workflow: Let the AI draft the questions, but always have a human teacher review and approve them before they are used.

Technical Summary: Small, Private Language Models as Teammates for Educational Assessment Design

Problem Statement
While Large Language Models (LLMs) have demonstrated capability in generating educational assessment questions aligned with pedagogical frameworks like Bloom's taxonomy, significant gaps remain in their evaluation and deployment. Current research often relies on subjective rubrics, limited statistical analysis, or proprietary cloud-based models, raising concerns regarding reproducibility, reliability, and real-world deployability. Furthermore, educational institutions frequently face constraints related to data privacy, cost, and resource availability, making reliance on external APIs problematic. Although Small Language Models (SLMs) offer local, privacy-preserving alternatives, their effectiveness for assessment design tasks—specifically regarding content validity, readability, and cognitive alignment—remains underexplored. Additionally, the trustworthiness of using AI models as evaluators ("model-as-a-judge") for generated content is not well understood, with potential biases and inconsistencies relative to human expert ratings.

Methodology
The study employs a systematic, end-to-end experimental workflow to compare frontier LLMs and locally deployable SLMs across four Research Questions (RQs).

Data and Models: The study utilized 2,550 questions from a prior dataset [23] and generated 4,080 new questions.
- LLMs (Frontier): GPT-4, GPT-3.5, Palm-2, LLaMA-2, Mistral, and Gemini Pro.
- SLMs (Local/Open-weight): Granite 4, Phi-4 mini, LLaMA 3.2, Mistral Small 3.2, GPT-OSS, Phi-4, Deepseek R1, and Gemma 3.
- Configuration: Generation and evaluation temperatures were fixed at 0.8 and 0.0, respectively, to ensure direct comparability.
Prompt Strategy: Five Prompt Structures (PS) were used, ranging from zero-shot general generation (PS1) to incrementally specific prompts targeting graduate levels and specific demographics (PS2–PS5), incorporating system instructions and explicit cues for Bloom's taxonomy levels.
Evaluation Metrics: The study paired each RQ with objective, reproducible NLP metrics and statistical testing:
- RQ1 (Cognitive Complexity): Measured via Flesch-Kincaid Grade Level and Flesch Reading Ease scores to assess text complexity and alignment with target grade levels.
- RQ2 (Linguistic Intent): Measured via BERTScore (using GTR T5-large embeddings) to compute semantic similarity between prompts and generated questions, and within-topic consistency across Bloom's levels.
- RQ3 (Pedagogical Compliance): Assessed via a "Verb Check" to determine if generated questions used action verbs consistent with their targeted Bloom's taxonomy level.
- RQ4 (Trustworthiness of Evaluation): Evaluated model-as-a-judge reliability by comparing model outputs against human expert annotations for High Quality (HQ) and Skill Match, analyzing agreement patterns, bias, and calibration.

Key Results

Generation Quality (RQ1–RQ3):
- Complexity: Both LLMs and SLMs produced questions at a comparable baseline grade level (late high school/early undergraduate, $\mu \approx 14$ ). However, SLMs demonstrated greater control and systematic modulation of language complexity in response to prompt specificity, whereas LLMs showed higher variability.
- Intent and Consistency: SLMs maintained closer topical intent alignment and exhibited higher semantic stability (higher F1 scores) across Bloom's level transitions compared to LLMs, which showed more conceptual drift.
- Pedagogical Alignment: Both families showed strong verb compliance when prompts were explicit. However, SLMs maintained consistent high alignment even across lower-specificity prompts, whereas LLMs relied heavily on explicit instructional scaffolding to maintain alignment.
Evaluation Reliability (RQ4):
- Neither SLMs nor LLMs consistently replicated the balance of selectivity and skill awareness found in human experts.
- Model-based judgments exhibited significant variance: some models were overly lenient (risking low-quality item acceptance), while others were overly conservative (risking the removal of high-quality items).
- Evaluator reliability was found to be dependent on the specific model being judged, with some models showing preferential agreement with generators from the same family, indicating systematic bias.

Key Contributions

Systematic Benchmarking: The study provides the first systematic comparison of SLMs and LLMs for Automated Educational Question Generation (AEQG), demonstrating that smaller, local models can achieve competitive, and in some cases superior, pedagogical alignment and stability compared to frontier models.
Reproducible Metrics: It introduces a framework using objective, reproducible NLP metrics (readability, semantic similarity, verb compliance) to evaluate generation quality across linguistic, semantic, and pedagogical dimensions, moving beyond subjective human rubrics.
Evaluation of Model-as-a-Judge: The research thoroughly analyzes the reliability and bias patterns of using LMs as evaluators, revealing that while they can assist, they currently lack the consistency required for autonomous assessment.
Deployment-Aware Insights: The findings highlight the viability of SLMs for privacy-sensitive, resource-constrained educational environments, offering a sustainable pathway for on-device deployment.

Significance and Claims
The paper posits that language models should be viewed as bounded assistants rather than autonomous assessors in educational workflows. The findings support a "Human-in-the-Loop" (HIL) framework where models accelerate ideation and scaffold question construction, but human experts retain curatorial authority to ensure pedagogical integrity and fairness.

The authors claim that while SLMs offer a promising alternative for local deployment due to their lower computational demands and consistent adherence to constraints, the trustworthiness of AI as a standalone evaluator remains unsettled. The study underscores the necessity of human oversight to mitigate model-induced biases and ensure that generated assessments are accessible and valid. Ultimately, the work advances the field by examining the trade-offs between quality, reliability, and deployment feasibility, laying a foundation for responsible, curriculum-aware GenAI systems in education.