Relational Visual Similarity

1. Problem Statement

Current state-of-the-art visual similarity models (e.g., LPIPS, CLIP, DINO) primarily focus on attribute similarity. These models determine similarity based on shared perceptual features such as color, shape, texture, and semantic class. While effective for tasks like object recognition, they fail to capture relational similarity—the human ability to perceive similarity based on abstract logical structures, functional relationships, or transformations between visual elements, even when the surface appearances differ drastically.

• The Gap: Humans can recognize that a burning matchstick sequence is similar to a ripening banana sequence because both represent a "transformation over time," despite having no shared visual attributes. Current models cannot make this connection.
• The Challenge: How to measure and model visual similarity that relies on underlying relational logic rather than visible content, and how to bring such images closer together in representation space.

2. Methodology

The authors propose a framework called Relational Visual Similarity (RelSim), which leverages Vision-Language Models (VLMs) and a novel dataset construction pipeline to capture relational logic.

A. Dataset Construction (114k Image-Anonymous Caption Pairs)

Since no existing dataset captures relational similarity, the authors curated a new dataset through a three-step process:

1. Image Filtering: They filtered the massive LAION-2B dataset to identify "relationally interesting" images (those containing transferable relational structures rather than just static objects). A fine-tuned Qwen2.5-VL-7B-Instruct model was used to classify images, resulting in 114,000 high-quality candidates.
2. Anonymous Captioning: To abstract away specific objects and focus on logic, the authors manually curated 532 groups of images sharing the same underlying relational pattern (e.g., a group of images showing "transformation of a subject through time").
   • A frozen VLM was prompted with these groups to generate anonymous captions.
   • These captions replace specific object nouns with placeholders (e.g., {subject}, {object}) to describe the relational logic (e.g., "A {subject} stands out against a gray background" or "Transformation of {subject} over time").
   • This creates a dataset of {Image, Anonymous Caption} pairs where the caption encodes the abstract logic rather than the visual content.
3. Training Objective:
   • Visual Encoder ($f_V$): A VLM (Qwen2.5-VL-7B-Instruct) is used as the visual feature extractor. Unlike pure vision encoders, VLMs integrate world knowledge necessary for relational reasoning.
   • Text Encoder ($f_T$): A standard text encoder (all-MiniLM-L6-v2) encodes the anonymous captions.
   • Loss Function: The model is trained using InfoNCE contrastive loss. The goal is to maximize the cosine similarity between an image's visual embedding and its corresponding anonymous caption embedding, while minimizing similarity with other captions in the batch.

B. Model Architecture

• Backbone: Qwen2.5-VL-7B-Instruct.
• Training Strategy: The visual encoder is fine-tuned using LoRA (Low-Rank Adaptation) for 15k iterations. A learnable query token is appended to the image input to guide the extraction of relational features.
• Inference: To measure similarity between two images, the model extracts visual embeddings and computes their cosine similarity. The model implicitly learns to align images that would generate similar anonymous captions.

3. Key Contributions

1. New Concept: Defined and formalized Relational Visual Similarity as a distinct and measurable dimension of image similarity, complementing traditional attribute-based similarity.
2. Novel Dataset: Created a dataset of 114k {Image, Anonymous Caption} pairs, specifically designed to capture abstract relational logic and abstraction rather than surface semantics.
3. New Metric (RelSim): Developed a tuned metric that successfully captures relational similarity, outperforming all existing baselines.
4. Empirical Analysis: Demonstrated that current SOTA models (CLIP, DINO, LPIPS) fail to capture relational similarity, even when fine-tuned, highlighting a critical gap in visual computing.
5. Applications: Validated the utility of RelSim in Relational Image Retrieval and Analogical Image Generation.

4. Experimental Results

The authors evaluated RelSim against baselines (LPIPS, DINO, DreamSim, CLIP-I, and caption-based retrieval methods) using both automated and human evaluation.

• Automated Evaluation (GPT-4o Judge):

  • In a retrieval task, GPT-4o scored the relational similarity of retrieved images on a scale of 0–10.
  • Results: Existing metrics scored between 4.56 (LPIPS) and 5.91 (CLIP-I).
  • RelSim: Achieved the highest score of 6.77, significantly outperforming all baselines.
  • Ablation: Fine-tuning pure vision encoders (CLIP/DINO) with anonymous captions improved their scores slightly but still fell short of the VLM-based RelSim, proving that world knowledge (from the LLM) is essential for relational reasoning.

• Human Evaluation (User Study):

  • Participants were asked to choose which of two retrieved images was more relationally similar to a query.
  • Results: Users preferred RelSim over baselines in 42.5% to 60.7% of cases (depending on the baseline), with the highest preference against CLIP-I (59.7%). This confirms that humans perceive relational similarity and that RelSim aligns with human intuition.

• Qualitative Analysis:

  • Retrieval: RelSim successfully retrieved images with different objects but identical logic (e.g., finding a "banana ripening" sequence when querying a "match burning" sequence), whereas CLIP retrieved visually similar but logically unrelated images.
  • Generation: In analogical image generation tasks, RelSim was used to evaluate how well models preserved the "underlying idea" of an input image. Proprietary models (e.g., GPT-4o, Nano-Banana) achieved higher RelSim scores (0.82–0.84) than open-source models (0.71–0.74), indicating a gap in their ability to transfer deep relational structures.

5. Significance and Future Directions

• Bridging the Gap: This work bridges cognitive science and computer vision, demonstrating that human-like visual understanding requires reasoning about relationships, not just recognizing attributes.
• Applications: The metric enables new capabilities in creative search (finding inspiration based on logic rather than appearance) and analogical generation (transferring conceptual ideas across different domains).
• Limitations & Future Work:
  • The current dataset relies on manually curated groups (532 groups), which limits scalability and may introduce bias.
  • The anonymous captioning model can hallucinate or exhibit biases.
  • Future work aims to automate the expansion of relational logic groups and develop methods to handle multiple valid relational mappings for a single image.

In conclusion, RelSim represents a paradigm shift from "what an image looks like" to "what an image represents logically," offering a crucial step toward more human-like visual intelligence.