🤖 AI

Human-in-the-Loop Benchmarking of Heterogeneous LLMs for Automated Competency Assessment in Secondary Level Mathematics

This paper introduces a Human-in-the-Loop benchmarking framework using a multi-dimensional rubric to evaluate heterogeneous LLMs for automated secondary-level mathematics assessment in Nepal, revealing that architectural compatibility with instruction constraints is more critical than model scale for achieving agreement with human experts and concluding that these models are best suited for assistive evidence extraction rather than autonomous certification.

Original authors: Jatin Bhusal, Nancy Mahatha, Aayush Acharya, Raunak Regmi

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jatin Bhusal, Nancy Mahatha, Aayush Acharya, Raunak Regmi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a classroom where teachers are drowning in paperwork. Instead of just giving students a single number (like "85%"), they want to give a detailed report card that says, "You understand the why," "You are great at calculating," and "You can connect ideas." This is called Competency-Based Education. It's a much richer way to grade, but it's incredibly hard and time-consuming for humans to do manually.

This paper asks a simple question: Can Artificial Intelligence (AI) help teachers do this heavy lifting?

Here is the story of their experiment, explained simply:

1. The Setup: The "Human vs. Robot" Test

The researchers in Nepal decided to test this on Grade 10 Math (a tough subject involving matrices, geometry, and trigonometry).

The Students: They took 33 handwritten math tests from real students.
The Human Judges: Two expert math teachers graded these tests. They didn't just look for the right answer; they used a very strict "rulebook" (rubric) to judge four specific skills:
1. Comprehension: Did they understand the question?
2. Knowledge: Did they know the facts?
3. Fluency: Could they do the math steps correctly?
4. Behavior/Correlation: Could they connect different ideas?
The AI Judges: They lined up four different AI models to grade the same tests using the same rulebook.
- Eagle: A small, fast AI (like a sprinter).
- Orion: A massive, powerful AI with huge brain power (like a heavyweight champion).
- Nova: A smart, efficient AI that uses a special "team" structure (like a specialist team).
- Lyra: A top-tier AI used as a referee.

2. The Twist: Bigger Isn't Always Better

Usually, we assume the biggest, most powerful AI (Orion) would win. But the results were surprising, like a giant sumo wrestler tripping over a small pebble while a nimble dancer glided past.

The "Giant" (Orion) Failed: Despite having the most "brain power" (70 billion parameters), Orion got confused. It disagreed with the human teachers so much that the math score for their agreement was actually negative. It was like a robot that tried to grade a test but invented its own rules that made no sense.
The "Specialist" (Nova) Won: The smaller, more efficient AI (Nova) did the best job. It agreed with the human teachers about 38% of the time (which is considered "Fair" in this strict world). It followed the instructions better than the giant.

The Lesson: In this specific task, following the rules was more important than having a big brain. The massive AI got "distracted" by its own complexity, while the specialized AI stuck to the job.

3. The Solution: The "Human-in-the-Loop"

The paper concludes that we shouldn't let the AI take the teacher's job entirely yet. The AI isn't ready to be the final judge.

Instead, think of the AI as a highly efficient intern.

The Intern (AI): Quickly scans the student's work, highlights the good parts, and suggests a grade based on the rulebook.
The Boss (Human Teacher): Looks at the intern's suggestions, double-checks the tricky parts, and makes the final decision.

This "Human-in-the-Loop" approach means the AI does the heavy lifting of finding evidence, but the human keeps the "credibility shield" to ensure fairness.

4. What This Means (and Doesn't Mean)

What it IS: A proof that AI can help teachers spot patterns and extract evidence from student work, making the grading process faster and more detailed.
What it IS NOT: A system ready to replace teachers or issue final certificates on its own. The paper explicitly warns that if we let AI grade alone, it might make mistakes (hallucinations) or be biased because we don't always know how it reached its conclusion (the "Black Box" problem).

In a nutshell: The researchers built a bridge between old-school grading and the future. They found that while AI isn't the captain of the ship yet, it makes a fantastic navigator, as long as a human is still holding the wheel.

1. Problem Statement

The paper addresses the bottleneck in transitioning from traditional marks-based assessment to Competency-Based Education (CBE) in secondary mathematics.

The Challenge: CBE requires qualitative mapping of student skills (e.g., comprehension, operational fluency) rather than simple numerical scores. This process is labor-intensive, subjective, and difficult to scale manually.
The Gap: While Large Language Models (LLMs) show promise in automated grading, there is a lack of empirical validation comparing LLMs against human experts at the competency-rubric level, particularly in school-level mathematics. Existing studies often suffer from the "Black Box" problem (lack of transparency), context limitations, and a lack of Randomized Controlled Trials (RCTs) against human ground truth.
Specific Context: The study focuses on the Grade 10 Optional Mathematics curriculum in Nepal, covering complex topics like Matrices, Coordinate Geometry, Trigonometry, and Functions.

2. Methodology

The authors employed a Human-in-the-Loop (HITL) benchmarking framework to evaluate the effectiveness of heterogeneous LLMs in automating competency assessment.

A. Dataset and Ground Truth Construction

Data Source: 33 handwritten response scripts from Grade 10 students in Nepal.
Assessment Design: 16 open-ended questions across four mathematical topics. Open-ended formats were chosen to capture reasoning steps and partial misconceptions.
Human Ground Truth: Two senior mathematics faculty members performed double-blind grading on the scripts.
- Inter-Rater Reliability: Achieved a Quadratic Weighted Cohen's Kappa ( $\kappa_w$ ) of 0.8652, indicating "Almost Perfect" agreement, establishing a robust ground truth.
- Adjudication: A third senior expert resolved any conflicts to finalize the ground truth labels.

B. Competency Framework

The study utilized a multi-dimensional rubric assessing four Cross-Cutting Competencies across four proficiency levels (Awareness, Application, Mastery, Influence):

Comprehension: Understanding the problem intent.
Knowledge: Conceptual depth and prerequisite knowledge.
Operational Fluency: Procedural accuracy and application of rules.
Behavior & Correlation: Strategic selection of tools and integration of concepts.

C. LLM Ensemble and Architecture

A multi-model ensemble was benchmarked against the human ground truth. All models used a uniform temperature (0.1) and a singular Master Evaluation Prompt (Role-Based Instruction: "Expert Mathematics Educator") to ensure consistency.

OCR/Digitization: Gemini 2.5 Flash (Multimodal) for transcribing handwritten scripts.
EAGLE: Llama 3.1-8B (Dense) – Optimized for speed/latency.
ORION: Llama 3.3-70B (Dense) – Optimized for deep reasoning.
NOVA: Gemini 2.5 Flash (Sparse MoE) – High-throughput control.
LYRA: Gemini 3 Pro (Sparse MoE) – "Judicial Arbiter" using Chain-of-Thought (CoT) to resolve inconsistencies.

D. Evaluation Metric

The primary metric was Quadratic Weighted Cohen's Kappa ( $\kappa_w$ ). This was chosen over simple accuracy because it accounts for the ordinal nature of the competency levels (e.g., penalizing a "Mastery" vs. "Influence" error less severely than a "Mastery" vs. "Awareness" error).

3. Key Contributions

Competency-Aligned Framework: Development of a specific rubric for Grade 10 Math (Nepal) that maps 16 topic-specific questions to four global cross-cutting competencies.
Human-LLM Benchmarking Protocol: A rigorous protocol involving double-blind human grading and a multi-model LLM ensemble evaluation, establishing a new standard for empirical validation in educational AI.
Architectural Analysis: An empirical investigation into how model scale (parameter count) vs. architecture (Dense vs. Sparse MoE) impacts performance in rubric-constrained tasks.
Open-Source & Proprietary Comparison: Direct comparison of open-weight models (Llama series) against frontier proprietary models (Gemini series).

4. Key Results

The study revealed a significant "Architecture-Compatibility Gap":

Human-Human Baseline: $\kappa_w \approx 0.87$ (Almost Perfect).
LLM vs. Human Performance:
- NOVA (Gemini 2.5 Flash): Achieved the highest agreement with humans ( $\kappa_w \approx 0.38$ ), classified as "Fair Agreement."
- LYRA (Gemini 3 Pro): Moderate alignment ( $\kappa_w \approx 0.27$ ).
- EAGLE (Llama 3.1-8B): Low alignment ( $\kappa_w \approx 0.10$ ).
- ORION (Llama 3.3-70B): Negative Agreement ( $\kappa_w = -0.0261$ ), indicating "No Agreement." The 70B model systematically contradicted human reasoning.
Inter-Model Consensus:
- Strongest agreement was between the two Gemini models (Lyra and Nova, $\kappa \approx 0.56$ ), suggesting a shared internal logic in the Sparse MoE architecture.
- Llama and Gemini models showed "Slight" to "Slight" agreement, indicating fundamental differences in interpreting mathematical data.

Critical Finding: Model compliance with instruction constraints and architectural fit (Sparse MoE) proved more critical for rubric-based tasks than raw parameter scale. The larger dense model (Orion) failed to adhere to the strict rubric, likely due to "instruction drift," while the smaller, optimized models performed better.

5. Significance and Conclusion

Paradigm Shift: The results challenge the assumption that "bigger is better." In specialized, rubric-constrained educational tasks, architectural compliance and instruction following outweigh raw parameter count.
Role of AI: LLMs are not yet suitable for autonomous certification due to the risk of hallucination and lack of deep pedagogical reasoning. However, they are highly effective as "First-Pass" Assistive Tools.
Human-in-the-Loop Value: The study validates a HITL framework where LLMs extract preliminary evidence and map competencies, significantly reducing the workload for human experts while maintaining the "credibility shield" of human oversight.
Scalability: While tested in Nepal, the competency framework (Comprehension, Knowledge, Fluency, Behavior) is universal, suggesting the methodology can be scaled to international contexts for more scientific and efficient assessment.

Future Directions: The authors recommend "Reasoning Audits" to verify AI logic, expanding sample sizes, and implementing a "Critic Loop" (two-AI system) to mitigate bias before full autonomous deployment.