Technical Summary: XPASS-VIS: A Dataset for Cross-Domain Personalized Image Aesthetic Assessment

1. Problem Statement

Personalized Image Aesthetic Assessment (PIAA) aims to model the subjective nature of aesthetic judgments at the individual level, moving beyond population-level consensus. While aesthetic preference is known to be deeply personal, it also exhibits consistency across visual domains; for instance, an individual's preference for simplicity may manifest in both minimalist art and understated fashion. However, existing PIAA datasets and methods are largely confined to single domains or lack sufficient per-annotator samples within each domain to support cross-domain analysis. Consequently, the extent to which personalized aesthetic preferences can be transferred across visual domains remains unexplored. Furthermore, collecting sufficient labeled data for every new domain is prohibitively expensive, creating a "cold-start" problem where no user rating history exists. This necessitates the development of Unsupervised Domain Adaptation (UDA) strategies to transfer models from labeled source domains to unlabeled target domains.

2. Methodology

2.1 The XPASS-Vis Dataset

To address the lack of data infrastructure for cross-domain PIAA, the authors introduce XPASS-Vis, the first dataset explicitly designed for this task.

• Stimuli: The dataset comprises 6,526 stimuli across three distinct visual domains: Artwork (2,345 images from the LAPIS test split), Fashion (2,082 high-resolution images from the Clothing Co-Parsing dataset), and Landscape (2,099 one-minute video clips from Sekai-Real-Walking-HQ, converted to single frames).
• Annotators: 129 annotators rated stimuli across all three domains.
• Annotations: Each annotator provided 10 ratings per stimulus:
  1. Overall Aesthetic Assessment: A 7-point Likert scale (1 = Highly unaesthetic, 7 = Highly aesthetic).
  2. Aesthetic-Emotion Ratings: Nine items from the AESTHEMOS questionnaire (e.g., "Liked it," "Made me feel nostalgic"), rated on a 5-point scale.
• User Attributes: Annotators provided demographics (age, gender, nationality, academic history) and personality traits (Ten-Item Personality Inventory/TIPI).
• Scale: The dataset yields 87,836 user–stimulus interactions, with each annotator rating over 200 stimuli per domain, ensuring sufficient coverage for personalization.

2.2 Experimental Setup and Baselines

The authors established a benchmark for cross-domain PIAA under an Unsupervised Domain Adaptation (UDA) setting, where a model is trained on a labeled source domain and transferred to an unlabeled target domain without using target labels.

• Preprocessing:
  • User Attributes: Encoded as one-hot vectors (114 dimensions total), including demographics, personality, and domain interest.
  • Image Attributes: 45 low-level Quantitative Image Properties (QIPs) extracted via a toolbox, covering color, texture, symmetry, and complexity.
• Training Pipeline:
  • General Image Aesthetic Assessment (GIAA): A baseline model (NIMA with CLIP backbone) trained to predict population-aggregated scores.
  • Personalized Image Aesthetic Assessment (PIAA): A two-phase approach:
    1. Pre-training (Population-level): A personalization module is trained on general users to learn population-level preference patterns.
    2. Fine-tuning (Individual-level): The model is fine-tuned for specific target users using a small number of their ratings (100 samples).
  • Baselines: The study evaluates proprietary Large Language Models (GPT-5.4, Claude Opus 4.6, Gemini 3.0 Flash) via zero-shot inference and two established PIAA architectures (MIR and ICI).
• Domain Adaptation Methods: The authors systematically evaluated 8 representative UDA methods:
  • Adversarial: DANN, CDAN, ALDA.
  • Discrepancy-based (Optimal Transport): DeepJDOT, JUMBOT.
  • Discrepancy-based (Feature Alignment): DeepCORAL, RSD, DARE-GRAM.
  • Note: RSD and DARE-GRAM are regression-aware variants specifically designed for continuous score prediction.

2.3 Evaluation

Performance was measured using 5-fold cross-validation at the user level. Metrics included:

• SCC (Spearman's Rank Correlation Coefficient): Measures rank-order correlation.
• CCC (Concordance Correlation Coefficient): Measures both rank-order and absolute agreement (sensitive to rating scale usage).

3. Key Results

3.1 Within-Domain Performance

• PIAA vs. GIAA: Personalized models (ICI and MIR) significantly outperformed the population-level GIAA model (NIMA) and zero-shot LLMs. The ICI model achieved the best overall performance (Avg. SCC = 0.470, CCC = 0.423).
• Domain Difficulty: Performance varied by domain, with Art yielding the highest scores and Fashion the lowest. This correlates with lower inter-annotator agreement (ICC) in fashion, suggesting higher subjectivity and difficulty in modeling population consensus for this domain.
• LLM Limitations: Zero-shot proprietary LLMs performed comparably to each other but failed to surpass the NIMA baseline in any domain, indicating that individual-level judgment is difficult to infer from text attributes alone without training data.

3.2 Cross-Domain Transfer (UDA)

• Domain Shift: The "Source-Only" baseline (no adaptation) performed poorly (Avg. SCC = 0.145), highlighting a substantial domain shift.
• Adaptation Success: Most UDA methods improved upon the Source-Only baseline.
  • Feature Alignment vs. Adversarial: Feature-alignment methods (DeepCORAL, RSD, DARE-GRAM) outperformed adversarial and optimal-transport methods.
  • Regression Awareness: The best-performing methods were RSD and DARE-GRAM, which are explicitly designed for regression tasks. They achieved the highest SCC scores among all UDA methods.
• Transferability: The best adaptation methods recovered approximately 60% of the Target-Only upper bound (the theoretical limit if target labels were available) in terms of SCC. This provides evidence that personalized aesthetic preferences are transferable across domains to a meaningful extent.
• Absolute vs. Relative Performance: While absolute scores were lower for fashion targets, the recovery rate relative to the target domain's difficulty was consistent across most directions, suggesting the adaptability of preferences is not direction-dependent but rather constrained by the inherent difficulty of the target domain.
• CCC Limitations: Recovery for CCC (absolute agreement) was lower (~38%) than for SCC, indicating that unsupervised adaptation struggles to capture annotator-specific rating tendencies (e.g., scale usage).

3.3 User-Level Analysis

The authors investigated whether specific user attributes (demographics, personality, rating style) predict who benefits from domain adaptation.

• Finding: Apart from a mechanical "regression-to-the-mean" effect, no user attribute systematically predicted adaptation gains. Significant effects were rare and confined to specific transfer directions.
• Implication: The benefits of marginal-alignment UDA are broad-based and not gated by coarse demographic or personality subgroups. Success appears driven by finer, intra-individual properties (e.g., the degree of preference structure overlap between domains) rather than static traits.

4. Key Contributions

1. XPASS-Vis Dataset: The release of the first dataset explicitly designed for cross-domain PIAA, featuring 129 annotators rating 6,526 stimuli across Art, Fashion, and Landscape with sufficient per-domain coverage (>200 samples/user) and rich user attribute collection.
2. Benchmark and Baselines: The establishment of a new benchmark task for cross-domain PIAA, including a systematic evaluation of 8 representative UDA methods and proprietary LLMs.
3. Empirical Evidence of Transferability: The demonstration that personalized aesthetic preferences are transferable across visual domains, with regression-aware feature-alignment methods recovering ~60% of the supervised upper bound in fully unsupervised settings.
4. Insight into Adaptation: The finding that adaptation gains are not predictable by standard user attributes, suggesting that future PIAA-specific adaptation strategies must focus on modeling the cross-domain preference structure itself rather than relying on demographic proxies.

5. Significance and Claims

The paper claims that XPASS-Vis provides a foundational resource for studying cross-domain PIAA, addressing a critical gap in existing literature where datasets were either single-domain or lacked per-user density. The results offer "encouraging evidence" that aesthetic preferences are not entirely domain-specific but contain transferable, domain-invariant components.

The authors modestly frame their findings as a "qualified yes" to the question of transferability. They emphasize that while current unsupervised methods recover a significant portion of performance, a substantial gap remains, particularly in capturing absolute rating tendencies (CCC). The paper posits that the next step for the field is the development of PIAA-specific adaptation strategies that explicitly model the interplay between user attributes and domain shifts, rather than relying on general-purpose UDA techniques. The work also highlights the need for future research into cross-modal adaptation and broader visual domains.