💬 NLP

How effective are VLMs in assisting humans in inferring the quality of mental models from Multimodal short answers?

The paper introduces MMGrader, a framework using concept graphs to infer the quality of students' STEM mental models from multimodal short answers, but finds that current vision-language models achieve only ~40% accuracy and thus fall short of human-level performance despite their potential to assist teachers in designing targeted pedagogical interventions.

Original authors: Pritam Sil, Durgaprasad Karnam, Vinay Reddy Venumuddala, Pushpak Bhattacharyya

Published 2026-03-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pritam Sil, Durgaprasad Karnam, Vinay Reddy Venumuddala, Pushpak Bhattacharyya

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading a stack of math and science homework. Usually, you just check if the final answer is right or wrong, like a traffic light: Green (Correct) or Red (Wrong).

But this paper asks a deeper question: "Do you really understand why the answer is right, or did you just get lucky?"

To answer this, the researchers are trying to build a digital assistant that can read a student's mind. They call these mental blueprints "Mental Models."

Here is a simple breakdown of their study, using some everyday analogies.

1. The Problem: The "Black Box" of Learning

Think of a student's brain as a black box. When they solve a physics problem about vectors (arrows showing direction and force), they might get the right answer, but their internal logic could be a mess.

The Old Way: Teachers use rubrics (checklists) to grade answers. It's like a human trying to guess what's inside the black box by looking at the outside. It's slow, tiring, and hard to do for a whole class.
The New Goal: The researchers want to peek inside the black box to see the Mental Model. Is the student's internal map of the world accurate, or is it full of holes?

2. The Solution: MMGrader (The "Mind-Map" Detective)

The researchers created a system called MMGrader. Think of it as a detective that tries to reconstruct the student's mental map.

The Blueprint (Concept Graph): Before the detective starts, they have a perfect "Master Blueprint" of how the topic should be understood. This is called a Concept Graph. It's like a subway map showing how different ideas (like "direction" and "magnitude") are connected.
The Clues (Multimodal Answers): Students don't just write words; they draw diagrams, scribble arrows, and write equations. The system has to read both the text and the drawings.
The Job: The system looks at the student's messy homework and tries to see: "Does this student's drawing match the Master Blueprint? How strong is their connection between these two ideas?"

3. The Experiment: Can AI Do the Detective Work?

The researchers asked a simple question: "Can current AI models (called Vision-Language Models or VLMs) act as good detectives?"

They took 9 different AI models (some small, some big) and gave them the same homework assignments that human experts had already graded. The AI had to look at the handwritten notes and diagrams and guess the "strength" of the student's understanding.

The Results: The AI is still a "Rookie Detective"

Human Experts: The gold standard. They got it right almost every time.
The Best AI (Molmo): It got about 40% accuracy.
- The Analogy: Imagine a game where you have to guess a number between 1 and 5. If the human says "4," the AI usually guesses "3" or "5." It's in the ballpark, but it's not precise.
The Average AI: Many models were confused. Some just guessed random numbers. Others got so confused by the handwriting that they started "overthinking" (talking to themselves in the output, like "Wait, is this an arrow or a letter?").

4. Why is this hard for AI?

The paper found that AI struggles with two main things:

Handwritten Chaos: AI is great at reading printed text, but student handwriting is messy. It's like trying to read a doctor's prescription written in crayon.
Deep Reasoning: The AI can see the picture, but it struggles to understand the logic behind it.
- Example: One AI looked at a diagram of vectors and started doubting itself: "Wait, maybe this is a unit vector? No, maybe it's polar coordinates?" It got stuck in a loop of confusion instead of just grading the answer.

5. The Future: Why Bother?

Even though the AI isn't perfect yet, the researchers believe it's a huge step forward.

The Vision:
Imagine a teacher with 30 students. Instead of spending 5 hours grading, the AI acts as a teaching assistant.

It doesn't just give a score of "B."
It tells the teacher: "Hey, 80% of your class understands the 'direction' of vectors, but they are all confused about 'magnitude.' Let's spend tomorrow's class fixing that specific hole in their mental map."

The Bottom Line

This paper is a reality check. It shows that while AI is getting better at reading our handwriting and understanding our drawings, it isn't quite ready to replace the human teacher's intuition yet.

The AI is currently like a student who is smart but nervous—it knows the concepts, but it gets tripped up by messy handwriting and complex logic. However, with a little more training, it could become the ultimate tool to help teachers understand exactly what their students are thinking, turning "grading" into "genuine understanding."

1. Problem Statement

The paper addresses the challenge of assessing STEM (Science, Technology, Engineering, Mathematics) mental models from student responses.

Context: Mental models are "units of coherently structured knowledge" representing a student's conceptual understanding. Assessing them is crucial for evaluating deep learning, not just surface-level correctness.
Challenge: Inferring these models from multimodal (text + diagrams) handwritten student answers requires deep reasoning and specialized expertise.
Limitations of Current Methods:
- Traditional rubric-based grading is labor-intensive and lacks scalability.
- Existing NLP/AI approaches often focus on short textual answers, semantic similarity, or simple labels (high/low), failing to capture the structured nature of mental models or handle multimodal inputs (diagrams, symbols, handwritten text).
Research Question: Can open-source Vision-Language Models (VLMs) effectively approximate human annotators in inferring the quality of STEM mental models from multimodal handwritten responses?

2. Methodology: MMGrader

The authors propose MMGrader, a novel framework that uses Concept Graphs as an analytical tool to infer mental models.

A. Analytical Framework: Concept Graphs

Concept Hierarchy: Topics are organized into Broad Concept Areas (BCAs), Sub-concept Areas (SCAs), and fundamental Key Concept Links (KCLs).
Graph Structure: A concept graph $G(V, E)$ is defined where vertices ( $V$ ) are conceptual units and edges ( $E$ ) represent relationships (concept links).
Scoring Mechanism: Instead of grading the whole answer, the system assigns a strength score (1–5) to specific concept links based on the student's response. This score reflects the coherence and depth of the student's understanding of that specific concept.

B. The MMGrader Pipeline

Input:
- Student assessment (handwritten text and diagrams).
- A pre-defined Concept Graph for the topic.
- A mapping between specific questions and relevant concept links.
Step 1: Evaluation (VLM Task):
- The VLM is tasked with analyzing the question, the student's multimodal answer, and the specific concept link.
- The model outputs a strength score (1–5) indicating how well the student demonstrated understanding of that link.
Step 2: Generation:
- Scores for individual concept links are aggregated (averaged) across relevant questions.
- A final concept graph is constructed for the student, representing the strength of their mental model across the topic.

C. Dataset Construction

Source: 10 questions on "Addition and Resolution of Vectors" (11th-grade curriculum).
Participants: 6 students with varying proficiency levels.
Annotations: 6 expert human annotators (2–4 years experience) rated the answers. Discrepancies were resolved via consensus to create ground-truth strength scores.
Data Volume: 895 data points (Question–Concept Link–Student Answer triplets).
Preprocessing: Handwritten text and diagrams were processed using Google's Gemini (gemini-2.5-flash) for OCR, as it outperforms other models on handwritten benchmarks.

3. Experimental Setup

The study evaluated 9 openly available VLMs (including Molmo, Pixtral, InternLM, Gemma, LLaVa, Qwen, LLama-Vision, Granite, and the closed-source Gemini) across four prompting scenarios:

Base Scoring: No rubric; relies on pretraining.
Generic Scoring: A generic 1–5 rubric provided.
Detailed Scoring: A concept-link-specific rubric provided (most guided).
Chain-of-Thought (CoT): Models instructed to reason step-by-step before scoring.

Evaluation Metrics:

Exact Match Accuracy: Percentage of scores matching human annotations exactly.
RMSE (Root Mean Squared Error): Deviation from human scores.
EMD (Earth Mover's Distance): How closely the distribution of model scores matches human scoring patterns.

4. Key Results

Human-Level Gap: No VLM achieved human-level performance. The best-performing model, Molmo, achieved only ~40% accuracy with an RMSE of 1.1 units.
Top Performer (Molmo):
- Accuracy: 39.98% (Generic Scenario).
- RMSE: 1.1.
- EMD: 0.58.
- Reasoning: Molmo's superior performance is attributed to its training on educational data.
Other Models:
- Pixtral: Showed low EMD (0.22 in Generic), indicating its score distribution mimics humans well, but individual predictions had higher error (RMSE ~1.5).
- Qwen: Showed significant improvement with detailed prompts and CoT, suggesting it can leverage fine-grained reasoning cues despite lacking specific educational pretraining.
- LLaVa & Granite: Performed poorly. LLaVa tended to output only extreme scores (1 or 5), while Granite (a small 2B model) failed to interpret diagrams and produced random scores or repetitive text.
Qualitative Failures:
- Overthinking/Self-Contradiction: Models like InternLM often doubted the diagram or notation, leading to incorrect scores.
- Format Issues: Some models (e.g., Gemini) returned outputs in LaTeX boxed formats (\boxed{3}) despite instructions to output plain integers, complicating automated parsing.

5. Key Contributions

MMGrader Framework: A novel approach to infer STEM mental models from multimodal responses using concept graphs as an analytical framework.
Decomposition Strategy: Breaking down topics into "concept links" and evaluating responses against a predefined mapping and scoring scale.
Comprehensive VLM Evaluation: The first extensive benchmark of 9 open-source VLMs on the specific task of inferring mental models from handwritten, multimodal STEM answers.
Empirical Findings: Established that while current VLMs are not yet ready for autonomous grading (due to ~40% accuracy), they show promise as assistive tools for teachers.

6. Significance and Future Directions

Pedagogical Impact: While not yet perfect for full automation, these models can assist teachers in scaling the assessment of mental models across entire classrooms. This enables the identification of collective proficiency gaps, allowing for targeted lectures and interventions.
Scalability: The approach moves beyond simple grading to quantifying the structure of student understanding, which is vital for personalized education and Intelligent Teaching Assistants.
Limitations: Current models require further fine-tuning on real-world, handwritten educational datasets to improve accuracy and reduce reasoning errors (e.g., overthinking).
Ethical Note: The study adhered to ethical standards, with anonymized student and annotator data and prior permissions obtained.

Conclusion: The paper demonstrates that while VLMs currently fall short of human experts in inferring complex mental models from multimodal inputs, they are approaching a threshold where they can serve as powerful, scalable assistants to educators, provided their accuracy is further improved through domain-specific fine-tuning.