💬 NLP

CArtBench: Evaluating Vision-Language Models on Chinese Art Understanding, Interpretation, and Authenticity

This paper introduces CARTBENCH, a museum-grounded benchmark comprising four subtasks to evaluate Chinese art understanding, interpretation, and authenticity in vision-language models, revealing that while current models achieve moderate recognition accuracy, they significantly struggle with evidence-grounded reasoning, expert-style appreciation, and connoisseur-level authenticity discrimination.

Original authors: Xuefeng Wei, Zhixuan Wang, Xuan Zhou, Zhi Qu, Hongyao Li, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xuefeng Wei, Zhixuan Wang, Xuan Zhou, Zhi Qu, Hongyao Li, Yusuke Sakai, Hidetaka Kamigaito, Taro Watanabe

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a group of very smart, super-fast robots that can look at pictures and read text. We call them Vision-Language Models (VLMs). Right now, these robots are great at simple tasks, like saying, "That's a cat!" or "That's a red apple."

But what happens when you ask them to do something much harder? What if you put them in a museum and asked them to act like a curator or an art expert? Could they explain why a painting is beautiful? Could they tell the difference between a real masterpiece and a perfect fake?

This paper introduces CARTBENCH, a new "exam" designed specifically to test these robots on Chinese Art. Think of it as a specialized driving test for robots, but instead of driving cars, they are driving through the complex world of Chinese history, brushstrokes, and cultural meaning.

Here is a simple breakdown of what they did and what they found, using some everyday analogies:

1. The Problem: The "Tourist" vs. The "Expert"

Most current AI models are like tourists with a guidebook. They can point to a statue and say, "That's a dragon from the Tang Dynasty." That's good! But if you ask them, "Why does the brushwork here suggest it was painted by a specific master in the Ming Dynasty?" or "Is this a real painting or a very good copy?", they often get lost. They might guess confidently but be completely wrong.

The researchers wanted to know: Can these robots think like art experts, or are they just good at memorizing facts?

2. The Exam: CARTBENCH

To test this, the team built a benchmark (a test suite) called CARTBENCH. It's like a four-part obstacle course for the robots:

Part 1: The Detective (CURATORQA)
- The Task: The robot looks at a painting and answers questions. But here's the catch: it can't just guess. It has to point to the specific part of the image that proves its answer.
- The Metaphor: Imagine a detective looking at a crime scene photo. They can't just say "The butler did it." They have to say, "The butler did it because you can see mud on his shoes that matches the garden."
- The Result: The robots were okay at simple questions but terrible at connecting visual clues to historical facts (like guessing the time period based on the painting style).
Part 2: The Poet (CATALOGCAPTION)
- The Task: The robot has to write a long, structured appreciation of the art, divided into four sections: Background, Content, Artistic Features, and Overall Verdict.
- The Metaphor: It's like asking a robot to write a professional museum guidebook entry. It needs to sound like a human expert, not a Wikipedia summary.
- The Result: The robots could follow the format (they wrote four sections), but the content was shallow. They sounded like a student trying to sound smart, missing the deep cultural nuance and emotional depth of a real expert.
Part 3: The Creative Thinker (REINTERPRET)
- The Task: The robot has to give a fresh, creative interpretation of a famous painting, but it must stay grounded in what is actually visible.
- The Metaphor: Imagine a teacher asking a student to write a new ending to a classic story, but the student can't invent new characters that don't exist in the book.
- The Result: The robots struggled to be truly creative without making things up. They often got stuck in "safe" answers or hallucinated facts.
Part 4: The Forger Hunter (CONNOISSEURPAIRS)
- The Task: The robot is shown two paintings side-by-side. One is real, one is a fake. They look almost identical. The robot must pick the real one.
- The Metaphor: This is the "tough guy" test. It's like asking someone to spot a counterfeit $100 bill when the fake one is printed so well it looks perfect to the naked eye.
- The Result: The robots performed no better than random guessing (like flipping a coin). They couldn't spot the subtle differences in brushstrokes or ink flow that a human expert would catch.

3. The Big Reveal

The study tested 9 different AI models (including big names like Qwen, GPT, and Gemini). Here is what they found:

The "Smart" Illusion: Some models got high scores on the easy questions. It looked like they were geniuses.
The Reality Check: When the questions got harder—requiring them to link a visual style to a specific historical era or spot a fake—their performance crashed.
The Language Gap: Models that were specifically trained to be bilingual (Chinese and English) did slightly better than general English-only models, but they still weren't "experts."
The Authenticity Wall: The robots are currently terrible at spotting fakes. They rely on surface details (like "this looks detailed") rather than deep understanding (like "the brush pressure here is inconsistent with the artist's style").

4. Why Does This Matter?

If we start using AI to help manage museums, restore art, or even authenticate paintings, we need to know their limits.

Risk: If a robot confidently tells a museum that a fake painting is real, the museum could lose millions of dollars.
Goal: This paper isn't saying "AI is bad." It's saying, "AI is a great assistant, but it is not a replacement for a human art expert yet." We need to teach these robots to look deeper, not just at the surface.

In a Nutshell

CARTBENCH is a reality check for AI. It showed that while robots are great at naming things, they are still very bad at understanding the soul, history, and subtle details of Chinese art. They are like tourists who can read the map but haven't learned how to navigate the terrain. To get there, we need to teach them to "see" like experts, not just like cameras.

1. Problem Statement

Current Vision-Language Models (VLMs) are increasingly deployed as multimodal assistants, yet their evaluation remains dominated by Western-centric concepts and short-form recognition tasks (e.g., object detection, simple QA). There is a critical gap in evaluating expert-facing, culturally grounded interpretation for Chinese art.

The Gap: Chinese art appreciation requires linking observable visual cues (motifs, brushwork, seals) to historical context and stylistic periods, rather than relying solely on surface recognition.
The Challenge: Existing benchmarks focus on factual correctness or short-form QA, failing to test long-form structured appreciation, defensible reinterpretation, and the subtle "connoisseur-level" reasoning required for authenticity discrimination (distinguishing originals from visually similar imitations).
Consequence: Without rigorous evaluation, VLMs risk producing culturally misaligned, hallucinated, or unsupported interpretations, limiting their utility in cultural heritage documentation and access.

2. Methodology: CARTBENCH Construction

The authors introduce CARTBENCH, a museum-grounded benchmark constructed through a three-phase pipeline aligning Wikidata entities with authoritative Palace Museum (Beijing) catalog pages.

Data Construction Pipeline

Alignment: Retrieval of image-bearing Palace Museum objects from Wikidata, aligned with official catalog descriptions.
Curation: Expert-guided filtering to ensure high-quality, traceable data across five art categories (Painting, Calligraphy, Sculpture, Ceramics, Stationery) and multiple dynasties.
Task Instantiation: The curated pairs are transformed into four distinct subtasks:

The Four Subtasks

CURATORQA (Evidence-Grounded Recognition & Reasoning):
- Scale: 14,421 questions over 1,589 artworks.
- Format: Multiple-choice and True/False.
- Focus: Tests if models can answer curator-style questions using explicit visual evidence. Includes difficulty levels: P1 (visual-only) and P2 (visual + art knowledge, e.g., inferring dynasty from style).
- Types: Covers subject recognition, scene classification, composition, technique, iconography, and style-to-period inference.
CATALOGCAPTION (Structured Expert Appreciation):
- Scale: 86 artworks.
- Format: Long-form generation requiring a specific four-section schema: Background, Content, Artistic Characteristics, Overall Evaluation.
- Goal: Evaluate the ability to generate structured, expert-style appreciation grounded in visible details, avoiding generic praise.
REINTERPRET (Defensible Reinterpretation):
- Scale: 25 canonical works.
- Format: Open-ended generation of "innovative appreciation."
- Evaluation: Two-stage human rating. Stage 1 filters for plausibility (no hallucinations/misreadings). Stage 2 rates creativity on a 1–5 Likert scale across five dimensions: Novelty, Coherence, Evidence-based Reasoning, Elaboration, and Insight.
CONNOISSEURPAIRS (Authenticity Discrimination):
- Scale: 10 authentic–imitation pairs.
- Format: Binary choice (A or B) on visually similar pairs.
- Goal: A diagnostic stress test to see if models can detect subtle global consistency (brush-and-ink coherence) vs. local cues (seals/inscriptions) to distinguish originals from high-quality replicas.

Evaluation Protocols

Automatic Metrics: Exact-match accuracy for QA; ROUGE, BLEU, CIDEr, BERTScore, and embedding similarity for captions.
Human Evaluation: Expert raters (art historians/students) score long-form tasks using rubrics for evidence linkage, cultural appropriateness, and insight depth.
Statistical Rigor: Bootstrap confidence intervals, sign-flip permutation tests, and inter-annotator agreement (Fleiss' $\kappa$ , ICC) are used.

3. Key Contributions

CARTBENCH Benchmark: The first unified suite for Chinese art evaluation covering recognition, structured appreciation, reinterpretation, and authenticity diagnostics across multiple dynasties and categories.
Unified Evaluation Framework: Combines automatic metrics with rubric-based expert scoring to assess evidence linkage, long-form compliance, and interpretive defensibility.
Identification of Failure Modes: Reveals that high overall accuracy masks specific weaknesses in style-to-period inference and evidence linking, and that current models fail significantly at connoisseur-level authenticity tasks.

4. Key Results & Analysis

The authors benchmarked 9 representative VLMs (including Qwen, GLM, GPT-5, and Gemini families).

CURATORQA Performance:
- Overall: Strong models (e.g., Qwen3-VL-235B) achieve high accuracy (~84%).
- The Drop: Performance sharply declines on P2 (knowledge-involved) and QA5 (style-to-period inference). For example, Qwen3-VL-235B drops from ~89% on simple recognition to ~56% on style-to-period inference.
- Bilingual Advantage: Chinese-English bilingual models (Qwen) significantly outperform general-purpose multilingual models (GPT-5-mini) on art-historical inference, highlighting the need for domain-specific cultural grounding.
CATALOGCAPTION Performance:
- Gap to Experts: Models perform poorly compared to human experts. The best model (Qwen2.5-VL-72B) achieves only ~37% of the human KPI score.
- Semantic vs. Structural: Models show high embedding similarity (capturing coarse intent) but low lexical overlap and poor adherence to the specific four-section structure required for expert appreciation.
REINTERPRET Performance:
- Plausibility Gate: The primary bottleneck is avoiding hallucinations. Models like Qwen3-VL-235B pass the plausibility gate ~88% of the time, while others fail more frequently due to era misattribution or content misreading.
- Creativity: Among valid outputs, creativity scores are relatively uniform (3.5–3.7/5), suggesting models struggle to generate truly novel, defensible interpretations beyond standard narratives.
CONNOISSEURPAIRS Performance:
- Near-Chance: Models perform near random chance (40–60% accuracy) on distinguishing authentic works from imitations.
- Failure Mode: Errors are attributed to over-reliance on local cues (seals, inscriptions) and surface detail richness, rather than global brush-and-ink coherence and stylistic consistency.

5. Significance and Future Directions

Cultural Grounding: The paper demonstrates that "general" VLMs lack the specific cultural and historical reasoning required for expert-level art analysis.
Beyond Recognition: It shifts the evaluation paradigm from "what is this?" to "why is this significant, and is it authentic?"
Diagnostic Value: While the authenticity and reinterpretation tasks are small-scale (due to high expert curation costs), they serve as critical diagnostic tools revealing fundamental limitations in current VLMs' ability to handle subtle, global visual consistency.
Implications: Future progress requires stronger visual evidence attribution and better modeling of medium-specific style cues. The benchmark provides a necessary testbed for developing culturally grounded, evidence-faithful multimodal systems for heritage applications.

Limitations Noted: The diagnostic tasks (REINTERPRET, CONNOISSEURPAIRS) are small-scale and should not be used for definitive model ranking. The dataset is currently sourced from a single institution (Palace Museum), which may limit generalization to other curatorial styles.