Technical Summary: Learning Visual Representations for Compositional Analysis of Artworks and Photographs

1. Problem Statement

Visual composition—the deliberate arrangement of visual elements—is fundamental to conveying meaning, emotion, and aesthetic quality in artworks and photographs. However, it remains one of the least formalized dimensions of visual understanding. Current computational approaches face two primary challenges:

1. Semantic Bias: Existing models often conflate compositional structure with semantic content (e.g., recognizing a "person" rather than the "spatial arrangement" of the person), leading to representations that are difficult to interpret and generalize poorly across domains (e.g., from photography to artwork).
2. Lack of Human Alignment: Many methods rely on abstract feature representations from deep neural networks or bounding boxes derived from object detectors trained on natural images. These fail to mimic human perceptual grouping, where the visual system organizes scenes into coherent regions and reasons about their spatial relationships. Furthermore, existing approaches often struggle with interpretability and require large amounts of annotated data, which is scarce for artwork composition.

2. Methodology

The paper proposes a comparative study of two paradigms for learning compositional representations: a human-inspired approach and fine-tuned foundation models.

A. Human-Inspired Approach (Object-Centric Learning + Graph Attention)

This paradigm explicitly models the principles of perceptual grouping:

1. Object-Centric Decomposition (OCL): The method utilizes Slot Attention to decompose an image into a fixed number of "slots," where each slot corresponds to a semantically meaningful region. Unlike bounding box methods, this unsupervised approach generalizes across domains (photography and artwork) without requiring task-specific object detectors or segmentation labels.
2. Relational Modeling (GAT): A Graph Attention Network (GAT) is employed to model the relationships between these region-level slots. Each slot acts as a node, and the GAT learns to assign importance weights to the edges (relationships) between them. This allows the model to capture spatial arrangements and identify salient elements based on their relational influence.
3. Architecture: The final compositional embedding is formed by concatenating the slot representations with the output of the GAT. This architecture is designed to be interpretable, as the graph structure explicitly represents region relationships.

B. Fine-Tuned Foundation Models

The second paradigm involves fine-tuning large-scale, self-supervised foundation models (specifically Dinov2 variants) on large-scale compositional datasets. This approach leverages the representational power of pre-trained models but treats them as "black boxes" that do not inherently follow perceptual grouping principles.

Datasets

The study evaluates both paradigms on:

• PICD: A dataset of 49k photographs with 24 composition categories.
• APDDv2: A dataset of 10k paintings with attribute-level scores, including a composition and layout score.

3. Key Contributions

• Comparative Analysis: The paper provides a systematic comparison between explicit, human-inspired compositional modeling (OCL + GAT) and data-driven fine-tuning of foundation models.
• Interpretable Representation: It demonstrates that object-centric learning can generate structured, interpretable compositional embeddings that align with human perceptual reasoning, unlike the abstract features of standard foundation models.
• Cross-Domain Generalization: The study highlights the ability of unsupervised OCL models to generalize from natural images to artwork without domain-specific retraining, addressing a key limitation of bounding-box-based methods.
• Unified Framework: The proposed architecture supports multiple downstream tasks (composition prediction, image retrieval, and saliency detection) within a single framework without requiring additional supervision or specialized architectures for each task.

4. Results

The experiments yield nuanced findings regarding performance, interpretability, and data efficiency:

• Frozen Encoders: When feature encoders are frozen, the human-inspired OCL + GAT approach significantly outperforms frozen foundation models (Dinov2) on composition category prediction and downstream tasks like compositional retrieval and visual saliency detection. This suggests that explicit modeling of region relationships is crucial when data is limited or when semantic bias must be minimized.
• Fine-Tuning: When sufficient labeled data is available, fine-tuned Dinov2 models achieve superior performance in terms of accuracy and score prediction (PLCC/SRCC) compared to the frozen human-inspired approach. However, this comes at the cost of interpretability and requires significantly more trainable parameters (e.g., 86M vs. 990K for OCL + GAT).
• Task Specificity:
  • Category Prediction: Explicitly modeling region relationships is sufficient for high-accuracy composition category classification.
  • Score Prediction: Predicting global composition scores (a continuous value) requires richer representations that capture broader image properties (e.g., color, contrast) beyond just inter-region relationships. In this task, fine-tuned models generally outperform the structured approach.
• Downstream Tasks:
  • Compositional Retrieval: The OCL + GAT model retrieves images based on composition categories with high precision, whereas frozen foundation models are heavily biased toward semantic content. Fine-tuned models improve retrieval but still rely on the specific training objective.
  • Visual Saliency: The GAT structure allows for the direct derivation of saliency maps by summing outgoing edge weights, identifying salient regions that align with human eye-tracking data. This provides a level of interpretability (identifying why a region is salient) that black-box models cannot offer.

5. Significance and Claims

The paper claims that while large-scale foundation models can achieve state-of-the-art performance given sufficient data and a well-defined objective, they do not inherently capture the structural logic of composition. The significance of the work lies in demonstrating that:

1. Interpretability vs. Performance Trade-off: There is a clear trade-off between the interpretability and data efficiency of human-inspired models and the raw performance of fine-tuned foundation models.
2. Necessity of Explicit Modeling: For tasks requiring cross-domain generalization (e.g., applying photography-trained models to art) or interpretability (e.g., explaining why an image has a certain composition), explicitly modeling meaningful regions and their relationships is superior.
3. Data Efficiency: The human-inspired approach achieves competitive results with a fraction of the trainable parameters, making it a viable alternative when large annotated datasets are unavailable.

The authors conclude that a "well-defined notion of composition combined with sufficient training data" can unlock the potential of existing large-scale models, but it does not replace the need for architectures that explicitly model compositional elements to ensure robustness, interpretability, and generalization.