Technical Summary: EduArt – An Educational-Level Benchmark for Evaluating Art History Knowledge in Large Language Models

Problem Statement

While Large Language Models (LLMs) have achieved ceiling performance on general-purpose benchmarks like MMLU, these aggregate measures fail to reveal model behavior within specific disciplines. Existing art-focused evaluations suffer from two primary limitations: (1) they rely heavily on synthetically generated questions rather than human-authored educational materials, and (2) they rarely report item-level psychometric properties, making it impossible to distinguish between genuine knowledge gaps, task-format sensitivity, or poorly constructed items. Furthermore, current benchmarks often conflate visual recognition with visual reasoning, and single-format evaluations (particularly multiple-choice) may systematically overestimate a model's ability to perform tasks requiring content production and manipulation, which are central to art-historical scholarship.

Methodology

The authors introduce EduArt, a multimodal benchmark designed to evaluate art-historical knowledge and visual reasoning across diverse question formats and languages.

Dataset Construction

The dataset comprises 871 human-authored questions drawn from two distinct, pedagogically grounded sources:

1. MyZanichelli (Italian): 668 questions from a digital exercise platform used in Italian secondary education, specifically covering Pittura rinascimentale (Renaissance paintings).
2. AP Art History (English): 203 questions from the College Board's Advanced Placement examinations (2013–2024).

The questions span seven distinct formats:

• multiple_choice_radio (Single correct answer)
• multiple_choice_check (Multiple correct answers)
• true_false
• completion_closed (Fill-in-the-blank with a constrained word bank)
• completion_open (Fill-in-the-blank without constraints)
• positioning (Drag-and-drop correct words into text)
• select_errors (Identification of incorrect words in a text)

The dataset includes 261 distinct images associated with 436 questions. All items are protected by copyright; the public repository provides the extraction and evaluation pipeline code rather than the raw data.

Evaluation Framework

Twelve models from six provider families (OpenAI, Google, Anthropic, Qwen, Mistral, Meta) were evaluated. The evaluation occurred under two conditions:

1. Default (Answer-only): The model returns only the answer in a strict JSON schema.
2. Motivation: The model is required to provide a 2–4 sentence written justification grounded in art-historical knowledge or visual evidence.

Performance was measured using format-specific metrics (e.g., exact match for radio MCQs, F1 score for multi-select and error identification, element accuracy for positioning).

Psychometric Analysis

The study applies Classical Test Theory (CTT) to characterize the benchmark. Key metrics include:

• Difficulty ($p$): The proportion of models answering an item correctly.
• Discrimination ($r_{pb}$): The point-biserial correlation between item performance and total score, indicating how well an item distinguishes between strong and weak models.

A logistic regression was also employed to isolate the independent effects of question format, language, image presence, and model identity on accuracy.

Key Results

1. Psychometric Properties

EduArt demonstrates strong psychometric properties. The mean discrimination across 831 estimable items is 0.514, with 82.3% of items classified as "good discriminators" ($r_{pb} \geq 0.30$). Item difficulty is well-distributed, ranging from very easy to very hard, confirming the benchmark is neither trivial nor systematically impossible.

2. Format Saturation and Divergence

• Multiple-Choice Saturation: Six models exceeded 94% accuracy on multiple_choice_radio questions, with overlapping confidence intervals. This indicates that recognition formats alone are insufficient to distinguish frontier models, as they have saturated this task type.
• Format Sensitivity: Model performance varies drastically across formats. Models scoring >94% on MCQs can drop to 23.9% on completion_open (e.g., Claude Opus 4.6) and as low as 6.2% on select_errors (e.g., Claude Sonnet 4.6).
• Ranking Instability: No model ranks consistently across all seven formats. For 10 of the 12 models, the spread between their best and worst format performance exceeds 50 percentage points.

3. Independent Predictors of Accuracy

Logistic regression analysis reveals:

• Question Format: Non-MCQ formats generally yield significantly lower odds of a correct response compared to multiple_choice_radio. Select_errors is the most difficult format (OR = 0.045), followed by completion_closed (OR = 0.256).
• Image Presence: Contrary to the intuition that visual aids help, the presence of an image is associated with a lower probability of a correct answer (OR = 0.765) after controlling for other variables. This suggests that for these models, language-based knowledge drives performance more than visual processing, and adding visual information introduces complexity that degrades accuracy.
• Language: Italian questions are significantly harder than English ones (OR = 0.459), likely due to the specific distribution of question types and source material.

4. The Motivation Condition

Requiring a written justification affects accuracy in a model-family-dependent manner:

• Claude models generally improved or maintained performance when asked to justify answers.
• Gemini and GPT models predominantly saw a decrease in accuracy (e.g., Gemini 3.1 Flash Lite Preview dropped by 12.8 percentage points).
• This dissociation suggests that the ability to articulate reasoning is distinct from the ability to select the correct answer, and that forcing structured output can interfere with answer selection in certain model architectures.

Significance and Claims

The paper claims that EduArt provides a valid, non-trivial tool for evaluating multimodal LLMs in a specialized domain. Its primary contributions are:

1. Moving Beyond Synthetic Data: By using human-authored educational materials, the benchmark avoids the pitfalls of synthetic generation and provides verifiable ground truth.
2. Revealing Capability Dissociations: The study demonstrates that art-historical knowledge (the ability to recognize facts) and the ability to deploy that knowledge (generating, placing, or correcting content) are distinct capabilities. High performance on multiple-choice recognition does not predict performance on open-ended or error-correction tasks.
3. Critique of Single-Format Benchmarks: The authors argue that single-format benchmarks (especially MCQs) systematically overestimate the range of tasks models can reliably perform. In art history, where tasks require producing and manipulating content rather than selecting from fixed options, MCQ-based evaluations offer a misleadingly optimistic picture.
4. Prerequisite for Responsible Adoption: The authors conclude that mapping this fine-grained capability profile is a precondition for the responsible use of multimodal LLMs in art-historical scholarship. Knowing precisely which tasks a model handles reliably is essential before deploying these tools for cataloging, attribution, or teaching, as scholarly work rarely takes the form of selecting an answer from a fixed set.