💬 NLP

Human-Grounded Multimodal Benchmark with 900K-Scale Aggregated Student Response Distributions from Japan's National Assessment of Academic Ability

This paper introduces Gakucho, a large-scale multimodal benchmark derived from Japan's National Assessment of Academic Ability that features 900,000 aggregated student response distributions to enable direct, human-grounded evaluation of multimodal large language models on authentic K-12 educational tasks.

Original authors: Kyosuke Takami, Yuka Tateisi, Satoshi Sekine, Yusuke Miyao

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kyosuke Takami, Yuka Tateisi, Satoshi Sekine, Yusuke Miyao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to test how smart a new robot is. Usually, you might give it a made-up quiz or a textbook problem that looks perfect on a computer screen. But this paper says, "Let's try something different." Instead of a fake test, let's give the robot a real, actual middle school exam from Japan, the kind that 900,000 real students took.

Here is the simple breakdown of what the researchers did and found:

1. The "Real-World" Test Kitchen

The researchers took official exams from Japan's National Assessment of Academic Ability. These aren't clean, digital text files. They are messy, real-world documents full of:

Diagrams and Maps: Like a weather map with squiggly lines.
Speech Bubbles: Text inside cartoon-style bubbles.
Vertical Text: Japanese is often written top-to-bottom, which is tricky for computers.
Tables and Charts: Numbers arranged in grids.

Think of this like trying to read a recipe that is written on a crumpled napkin with a sketch of a pot next to it, rather than a clean digital PDF. The researchers had to carefully "translate" these messy pages into a format computers could understand without losing the original look and feel.

2. The Secret Ingredient: The "Human Crowd"

The coolest part of this project is that they didn't just keep the questions; they kept the answers from 900,000 real students.

Imagine you have a test question. Usually, you only know the "right" answer. Here, the researchers know exactly what percentage of students got it right, what percentage got it wrong, and how they got it wrong.

Why this matters: It lets the researchers compare the robot's brain directly against a giant crowd of human brains. They can see: "Did the robot make the same mistake a human would make, or did it make a weird robot mistake?"

3. Testing the Robots

They fed these real exams to several famous AI models (like GPT-4, Gemini, and Claude) to see how they did.

The Results:

Science: The robots were pretty good, especially when they could "see" the diagrams. It's like they finally understood that a picture of a volcano is important for the question.
Math: They were great at simple numbers but stumbled when the math required reading a graph or a weirdly drawn shape. If the robot couldn't "read" the picture, it couldn't solve the math.
Japanese (Language): This was the hardest. The robots struggled with text written vertically and with recognizing specific characters (Kanji) inside images. It's like asking a robot to read a handwritten note on a napkin; they often guessed or made up words instead of reading exactly what was there.

4. The "Judge" Problem

The researchers tried two ways to grade the robots:

Human Teachers: Actual people read the answers.
AI Judges: Another AI (GPT-4o) graded the first AI's answers.

The Twist:

For Science and Math, the AI Judge was a pretty good substitute for a human teacher. They mostly agreed on who passed and who failed.
For Japanese, the AI Judge was confused. It often disagreed with the humans. The paper suggests that for complex language tasks, you can't just trust an AI to grade another AI yet; you still need a human in the loop.

5. Where the Robots Stumbled

The paper highlights some funny but revealing failures:

The "Fake Quote" Problem: When the test asked the robot to "copy this sentence exactly from the image," the robot sometimes made up a sentence that sounded right but wasn't actually there. It hallucinated instead of reading.
The "Drawing" Problem: One question asked the robot to look at a specific Japanese character and explain why its visual balance was off. The robots couldn't "see" the tiny details of the drawing well enough to answer.

The Bottom Line

This paper built a new, realistic playground for testing AI. Instead of testing robots on clean, perfect data, they tested them on the messy, visual, and complex reality of a real school exam.

They found that while AI is getting smarter, it still struggles with the specific "visual logic" of real-world exams, especially when it comes to reading text inside images or understanding the subtle mistakes real students make. They also proved that you can't always trust an AI to grade another AI, especially in language subjects.

Where to find it: The researchers made all this data (the questions, the images, and the 900,000 student answers) available for anyone to use, so other scientists can run their own tests.

Technical Summary: Human-Grounded Multimodal Benchmark with 900K-Scale Aggregated Student Response Distributions

Problem Statement
While Multimodal Large Language Models (MLLMs) are increasingly evaluated using exam-style benchmarks, existing resources (e.g., MMLU, C-Eval) primarily rely on synthetic or curated textual data. These benchmarks fail to capture the authentic multimodal characteristics of real-world K–12 assessments, particularly within Japanese educational contexts. Japanese middle-school examinations feature complex, heterogeneous layouts integrating dense visual elements (diagrams, tables, maps), vertical text, and speech balloons within structured pedagogical designs. Current benchmarks lack the grounding in real-world educational data and, crucially, do not provide population-level response distributions. This absence makes it difficult to meaningfully compare model performance against human learners or to empirically estimate task difficulty and reasoning patterns. Furthermore, raw exam materials are often released as complex PDFs with irregular panel boundaries and implicit reading orders, presenting significant bottlenecks for machine processing, including OCR errors on vertical text and difficulties in segmenting uneven panels.

Methodology
The authors constructed a multimodal dataset derived from Japan's National Assessment of Academic Ability (NAAB), specifically utilizing officially released Grade 9 items from the 2022 administration in Science, Mathematics, and Japanese Language.

Data Construction Pipeline: The authors developed a pipeline to transform human-friendly but machine-unfriendly PDF exam booklets into a structured, reproducible benchmark.
1. Manual Transcription: Questions were converted to Markdown, preserving original reading order and inserting image references. Mathematical expressions were rendered in LaTeX.
2. Image Extraction: Diagrams and figures were extracted and linked to specific items, preserving the visual layout without adding annotations to the figures themselves.
3. Schema-based Conversion: A unified JSON schema was generated to link sub-questions to the specific panels they reference, normalizing reading order and cross-panel dependencies.
4. Response Metadata Integration: Crucially, the dataset incorporates nationwide aggregated student response distributions ( $N \approx 900,000$ ) extracted from official survey reports. This includes answer distributions, correct/incorrect examples, and scoring conditions.
Evaluation Framework: The benchmark was used to evaluate a suite of representative MLLMs, including API-accessible systems (e.g., GPT-5, GPT-4o, Gemini 2.5/3.1, Claude Sonnet 4/4.5) and open-weight models (e.g., Qwen2.5-VL, Gemma 3).
- Metrics: Exact-match (EM) accuracy was used for multiple-choice (MC) items, and character-level F1 scores were used for open-ended responses.
- Scoring Validation: Open-ended responses were evaluated via both human annotation (by two authors, one with K–12 teaching experience) and an "LLM-as-a-judge" approach using GPT-4o to assess scoring reliability and alignment.

Key Contributions

Authentic Multimodal Benchmark: The paper introduces the first benchmark derived from Japan's NAAB that preserves authentic exam layouts, diagrams, and Japanese educational text features, addressing the scarcity of resources grounded in Japanese K–12 assessments.
Population-Level Response Data: The dataset includes aggregated response distributions from approximately 900,000 students. This enables empirical estimation of task difficulty and allows for direct comparison between human and LLM performance under a unified evaluation framework.
Structured Multimodal Representation: The authors propose a method to transform complex exam PDFs into a structured JSON format that links sub-questions to visual panels, facilitating multimodal reasoning evaluation while maintaining the integrity of the original visual evidence.
Human-Model Comparison: By providing both human-graded baselines and LLM-as-a-judge analyses, the work supports the assessment of automatic scoring reliability and establishes a reproducible standard for cross-model comparison in educational reasoning.

Experimental Results

Subject-Specific Performance: MLLMs exhibited distinct behaviors across subjects. In Science, models like GPT-5 and Gemini 2.5 showed strong performance. In Mathematics, multiple-choice accuracy was relatively stable across models (averaging ~0.85), though open-ended F1 scores varied significantly. In Japanese Language, performance diverged; Gemini 2.5 achieved the highest MC accuracy (0.800), while Claude Sonnet 4 led in open-ended generation (0.342).
Visual Reasoning Sensitivity: Ablation studies indicated that images substantially improved performance in Science. In Math and Japanese, the impact of images was mixed, reflecting task-dependent reliance on visual information.
Error Analysis: Heatmaps revealed that while models performed well on simple numerical problems, they struggled with items requiring the interpretation of diagrams and graphs (e.g., reading histograms). In Japanese, models failed significantly on tasks requiring fine-grained visual analysis of kanji characters (e.g., stroke count and spatial proportioning) and exact transcription from images, often producing fabricated text instead of verbatim quotes.
Human vs. LLM-as-a-Judge: Correlation between human accuracy and GPT-4o judge accuracy was strong in Science ( $r=0.899$ ) and moderate in Math ( $r=0.600$ ). However, for Japanese, the correlation was near zero ( $r \approx 0$ ), attributed to restricted dynamic range, small sample sizes, and scoring skew, suggesting caution when using LLM-as-a-judge for Japanese open-ended tasks.

Significance and Claims
The paper positions this dataset as an initial step toward systematic benchmarking of multimodal comprehension in Japanese educational contexts. Its primary significance lies in bridging textual, visual, and cognitive reasoning while establishing reproducible standards for cross-model comparison. The authors claim that the inclusion of aggregated student response distributions enables a "human-grounded" evaluation, allowing researchers to determine if model errors align with real student misconceptions. The dataset is intended to support future research in evaluation, feedback generation, and explainable AI within authentic assessment contexts. The authors remain modest, noting that the current release is limited to a single year of data and a specific national context, and that the evaluation results serve as preliminary validation rather than definitive evidence of scoring reliability.