Visual Reasoning Benchmark: Evaluating Multimodal LLMs on Classroom-Authentic Visual Problems from Primary Education

1. Problem Statement

While Multimodal Large Language Models (MLLMs) have achieved state-of-the-art performance in textual reasoning and symbolic mathematics, they exhibit significant deficiencies in visual reasoning, particularly regarding spatial and relational structures.

• The Gap: Existing benchmarks (e.g., ARC, MathVista) often rely on synthetic puzzles or contain heavy textual scaffolding (linguistic cues) that allow models to bypass genuine visual inference.
• The Educational Need: Primary education, especially in foundational mathematics, relies heavily on "non-verbal reasoning" (interpreting diagrams, patterns, and spatial relationships without text). Current models lack the robustness to handle classroom-authentic visual problems, which often include unedited images with artifacts (e.g., photocopy noise) and minimal text.
• The Risk: Deploying AI tutors or grading systems that fail at these tasks could lead to incorrect marking, false scaffolding, and the reinforcement of student misconceptions.

2. Methodology

The authors introduced the Visual Reasoning Benchmark (VRB), a dataset designed to test MLLMs under realistic, low-resource educational conditions.

• Data Source:
  • 701 Multiple-Choice Questions (MCQs) extracted from primary school examinations in Zambia (National End of Primary Exams) and India (Jawahar Navodaya Vidyalaya Selection Test).
  • These sources represent authentic, high-stakes aptitude tests used for real student assessment.
• Data Processing:
  • Minimal-Text Approach: Images were cropped directly from PDFs without normalization, sharpening, or text removal. This preserves "noise" (artifacts, slight misalignments) common in Low- and Middle-Income Country (LMIC) settings.
  • Prompting: Models were evaluated using an "image-first, minimal-text" prompt strategy to isolate vision-centric reasoning, avoiding linguistic hints.
• Annotation & Validation:
  • Human Consensus: Three human markers (two education experts, one non-expert) independently answered and categorized each question.
  • Selection Criteria: Questions were retained only if there was majority consensus (2/3 or 3/3 agreement) on the answer. 34 questions were excluded due to ambiguity; 10 were removed due to fatal image errors.
  • Taxonomy: Questions were annotated by Task (6 categories: e.g., pattern completion, odd-one-out) and Skill (10 categories: e.g., counting, folding, rotation, scaling).
  • Quality Control: Images were tagged for error severity (No Error, Minor, Moderate) to test robustness against visual noise.

3. Key Contributions

1. VRB Dataset: The first large-scale, classroom-grounded benchmark for visual reasoning in LMIC contexts, featuring unedited, minimal-text images.
2. Definition of "Classroom Authenticity": Moving beyond synthetic puzzles to evaluate models on genuine student materials, including the imperfections found in real-world educational resources.
3. Establishment of a Capability Threshold: The authors define a 94% human consensus threshold as the minimum requirement for a model to be considered useful in a classroom setting.
4. Process-Aware Evaluation: The benchmark distinguishes between static skills (counting, scaling) and dynamic spatial operations (folding, rotation, reflection), revealing specific architectural bottlenecks.

4. Key Results

The paper evaluated 45 proprietary and open-weight MLLMs.

• Overall Performance:
  • Accuracy ranged from 23% (near random guessing) to 78%.
  • Even the top-performing models (Gemini-3.0 Flash/Pro at ~76-78%) fall significantly short of the 94% human threshold required for reliable classroom deployment.
  • Open vs. Proprietary: Proprietary models generally dominate. While some open-weight models (e.g., Kimi-K2.5 at 60%) show promise, they often require massive computational resources (trillion parameters) that negate the cost/accessibility benefits of local hosting in LMICs.
• The "Jagged Frontier":
  • Performance is highly inconsistent. Models excel at static skills (Counting, Scaling) but hit a "spatial ceiling" on dynamic operations.
  • Skill Analysis:
    • Positive Correlation: Counting (+1.4 pts) and Scaling (+5 pts) improve accuracy.
    • Negative Correlation: Complex spatial manipulations drastically reduce accuracy: Folding (-9 pts), Reflection (-5.8 pts), and Rotation (-5 pts).
  • Task Analysis: Models struggle most with "Match (process)" tasks requiring multi-step transformations, while "Odd One Out" (direct comparison) is easier.
• Robustness to Noise:
  • As image quality degrades (from pristine to moderate errors), model accuracy drops from 38% to 30%.
  • High-performing models are surprisingly brittle, relying on precise pixel matching that fails when artifacts disrupt low-level patterns. Lower-performing models show less variance because they fail to extract structural information regardless of quality.
• Cost vs. Performance:
  • A "Visual Value Frontier" was identified. Gemini-3 Flash offers the best trade-off (78% accuracy at $0.50/M tokens), outperforming models costing ten times as much.

5. Significance and Implications

• Safety in Education: The findings suggest that current MLLMs are not yet safe for unsupervised classroom use in visual reasoning. Their inability to handle dynamic spatial tasks and their brittleness to image noise pose risks of reinforcing student misconceptions.
• Architectural Limitations: The results confirm a "spatial gap" in current AI architectures. Models treat vision as a secondary cue to language priors and lack the internal mechanisms for "spatial imagination" (simulating 3D transformations). They struggle to maintain geometric identity during mental manipulation.
• Future Directions:
  • Human-in-the-Loop: Visual reasoning tasks in education require explicit human oversight.
  • Process-Aware Evaluation: Future benchmarks must evaluate reasoning fidelity (how the model solves the problem), not just the final answer, to distinguish between correct answers derived from flawed logic.
  • Tool Use: Integrating external tools (e.g., Python libraries for geometric processing) may help bridge the gap between perception and spatial reasoning.

In conclusion, the VRB establishes that while MLLMs are improving rapidly, they have not yet reached the reliability required for primary education visual reasoning, particularly in dynamic spatial tasks and real-world, noisy environments.