Technical Summary: ArtiFact: A Large-Scale Multi-Modal Cultural Heritage Dataset

Problem Statement

Multi-modal data management has become a central research frontier in the database community, encompassing data integration, semantic query processing, and data quality assessment. However, a significant gap exists in the availability of large-scale, real-world datasets that jointly combine tabular, textual, and image data. While existing benchmarks often assume metadata and images are consistent, real-world cultural heritage collections frequently contain noisy annotations, ambiguous terminology, missing information, and historical inconsistencies. Furthermore, existing large-scale collections (e.g., OmniArt, SemArt) often leave data quality and metadata correctness unexamined, and cross-modal error detection benchmarks are largely restricted to e-Commerce or web data, lacking representation in the cultural heritage domain.

Methodology

The authors introduce ArtiFact, a dataset comprising 651,045 museum records aggregated from three major institutions: the Metropolitan Museum of Art (MET), the Art Institute of Chicago (AIC), and the Rijksmuseum. The dataset generation involves a unified preprocessing and normalization pipeline (ETL) designed to handle heterogeneous metadata standards:

1. Data Collection: Records were harvested via REST APIs, JSON-LD files, and OAI-PMH protocols. Only records with public-domain image URLs were retained.
2. Preprocessing & Normalization:
   • Rule-based Processing: Dates were standardized to absolute bounds (e.g., "late 18th century" to 1767–1800), units were converted to metric (cm/g), and artist attributions were split into structured fields (name, role, nationality, dates).
   • LLM-assisted Parsing: Approximately 168,000 "hard" rows that failed rule-based parsing were processed using Google's Gemini 2.5 Flash Lite with Chain-of-Thought prompting to structure complex materials and techniques.
   • Schema Mapping: Data was consolidated into a unified 24-column schema covering identifiers, temporal, physical, cultural/geographic, and artist attributes.
   • Deduplication: Categorical values (materials, techniques) were embedded using all-MiniLM-L6-v2 and classified by an LLM to identify synonyms and hierarchical relationships. Artist deduplication was framed as entity matching based on five cleaned attributes.
3. Error Injection for Evaluation: To facilitate cross-modal error detection research, the authors curated a taxonomy of seven error categories (Physical, Culture, Temporal, Identity, Geographic, Spatial, Visual) and injected them into 130,209 records. These errors were designed to be plausible yet factually incorrect, such as swapping images of visually similar objects, shifting temporal ranges by centuries, or swapping geographic locations between neighboring countries.

Key Contributions

The paper presents three primary contributions:

1. The ArtiFact Dataset: A large-scale multi-modal dataset of 651,045 objects paired with tables, images, and text, available as an open-source resource.
2. A Unified Cleaning Pipeline: A reproducible pipeline for parsing and normalizing heterogeneous museum data using a combination of rule-based processing and LLM-assisted semantic parsing.
3. Empirical Evaluation: A benchmark evaluation of the dataset on two downstream tasks, demonstrating its utility as a testbed for multi-modal data management research.

Results

The authors evaluated the dataset using two downstream tasks:

1. Cross-Modal Error Detection

Using Gemini 3 Flash as a baseline on 4,000 samples (200 per error subtype plus clean records), the study revealed a clear difficulty spectrum:

• High Performance: Visually or semantically salient errors, such as image swaps (easy/medium/hard), 10x scale errors, and continent-level culture swaps, were detected with high precision and recall (F1 scores ranging from 0.89 to 0.96).
• Low Performance: Subtle, domain-specific inconsistencies remained challenging. Material anachronisms (F1 0.58), aspect ratio swaps (F1 0.53), and temporal shifts (F1 0.63) were frequently missed or misclassified.
• Observation: The system often misattributed errors; for instance, aspect-ratio errors were sometimes flagged as artist inconsistencies, and certain temporal errors were deemed historically plausible. This suggests that reliable detection requires deep domain knowledge and multi-modal reasoning.

2. Semantic Query Processing

The authors tested semantic query engines (LOTUS and Palimpzest) on five queries involving cultural proximity, ambiguous object types, and historical terminology.

• Performance: F1 scores ranged from 0.39 to 0.85.
• Key Challenges:
  • Object Ambiguity: Historical vases with figurative shapes caused false positives.
  • Cultural Proximity: Systems struggled to distinguish British alabaster objects from visually similar Dutch and German artifacts.
  • Terminology: Queries involving specific techniques (e.g., albumen silver printing) were often conflated with unrelated processes like dyeing or weaving.
  • Historical Nuance: Systems failed to resolve historically contingent material and technique terminology or account for the evolution of object appearances across time and geography.

Significance and Claims

The paper positions ArtiFact as a challenging benchmark for multi-modal data management research. The authors claim that while current systems can handle visually obvious inconsistencies, they fail significantly when dealing with subtle domain-specific nuances that require cultural and historical knowledge.

The significance of the work lies in:

• Addressing the scarcity of large-scale, well-curated multi-modal datasets in the cultural heritage domain.
• Providing a realistic testbed that exposes the limitations of current multi-modal models in handling noisy, heterogeneous, and historically complex data.
• Highlighting that reliable cross-modal error detection and semantic query processing in this domain require more than just statistical correlation; they necessitate the integration of domain expertise to handle cultural biases and historical contingencies.

The authors conclude that ArtiFact aims to stimulate further research into multi-modal data management, specifically focusing on the challenges posed by cultural and historical biases in data quality and query processing.