Technical Summary: SOCO – Benchmarking Semantic Object Correspondence in Vision Foundation Models

Problem Statement

Measuring structured object understanding in Vision Foundation Models (VFMs) remains a significant challenge due to inconsistent evaluation protocols and limited part-level supervision. While existing benchmarks cover category-level recognition (e.g., ImageNet) or spatial localization (e.g., detection, segmentation), they offer limited insight into whether a representation captures structured object understanding: the ability to relate semantically corresponding parts across different object instances and categories.

Current Semantic Correspondence (SC) research is constrained by:

1. Ambiguous Task Definitions: Existing benchmarks often conflate recognizing a local concept (e.g., a wheel center) with identifying its specific instance within an object (e.g., front-left vs. rear-right wheel).
2. Lack of Cross-Category Evaluation: Most datasets evaluate correspondence only within a single category, failing to assess the transfer of knowledge across related categories (e.g., matching a wheel on a car to a wheel on a bus).
3. Inconsistent Annotations: Keypoint definitions in prior datasets (e.g., SPair-71k, MISC210K) are often geometric heuristics rather than semantically grounded concepts, leading to ambiguity in symmetric objects or those with high intra-class variability.
4. Limited Multimodal Evaluation: Current benchmarks lack language descriptions, preventing systematic evaluation of Large Vision-Language Models (LVLMs) regarding their fine-grained part-level understanding.

Methodology: The SOCO Framework

The authors propose Semantic Object Correspondence (SOC), a taxonomy-driven formulation that disentangles three distinct abilities:

1. Concept Correspondence (CC): Matching the same local semantic concept (e.g., "wheel center") across instances, regardless of position.
2. Semantic Object Correspondence (SOC): Matching the same concept with the same object-relative identity (e.g., "front-left wheel"). This requires capturing both semantics and geometric position.
3. Cross-Category SOC (Cross-SOC): Matching object-relative keypoints across related categories via shared taxonomy concepts (e.g., matching a wheel on a car to a wheel on a tractor).

The SOCO Dataset

To operationalize this taxonomy, the authors introduce SOCO, a large-scale benchmark featuring:

• Scale and Diversity: 100 diverse categories organized into four super-classes (Transportation, Hand-held Objects, Furniture, Animals), totaling 4,000 images and over 1 million correspondence pairs.
• Taxonomy-Driven Annotations: Keypoints are defined as unique semantic concepts with disambiguating positional attributes (e.g., "front," "left," "rear") within an object-centric coordinate system. This resolves ambiguity in symmetric objects.
• Language Grounding: Every keypoint includes a human-specified language description (e.g., "Center point of the front left wheel of a bus"), enabling the evaluation of LVLMs.
• Data Sources: Images are sourced from ImageNet3D (for man-made objects) and Animal3D (for animals), ensuring valid 3D pose metadata.

Evaluation Protocols

• Vision Models: Evaluated in a zero-shot manner using nearest-neighbor feature matching (argmax cosine similarity) on frozen backbones. Performance is measured via Percentage of Correct Keypoints (PCK@0.1).
• LVLMs: Evaluated as a multiple-choice Visual Question Answering (VQA) task using the CircularEval protocol. Three settings are tested:
  1. Vis.: Visual query (marked source image).
  2. Vis.+Desc.: Visual query + textual description.
  3. Desc.: Textual query only (no source image).
• Downstream Correlation: The paper correlates SOC performance with dense downstream tasks (segmentation, tracking, 3D pose, 3D detection) and compares it against ImageNet kNN classification.

Key Contributions

1. Task Formulation: The introduction of Semantic Object Correspondence (SOC) as a principled decomposition of semantic correspondence into concept recognition, structured object understanding, and cross-category transfer.
2. Dataset Release: SOCO, the first benchmark combining a hierarchical keypoint taxonomy, language descriptions, and cross-category correspondence pairs across 100 categories.
3. Vision-Model Analysis: Extensive evaluation across a broad family of VFMs (DINO, CLIP, Stable Diffusion, I-JEPA, etc.) revealing distinct failure modes.
4. LVLM Analysis: A systematic evaluation of LVLMs showing a performance gap between text-prompted localization and visual-reference matching.
5. Diagnostic Utility: Demonstration that SOC performance correlates more strongly with dense downstream tasks than ImageNet classification accuracy.

Experimental Results

Vision Foundation Models

• Semantic vs. Geometric Awareness: Strong VFMs exhibit a consistent and substantial performance drop from Concept Correspondence (CC) to Semantic Object Correspondence (SOC). For example, DINOv2 drops from 78.9% (CC) to 60.4% (SOC). This indicates that while models encode strong local semantic features, they struggle to disambiguate repeated parts (e.g., left vs. right wheels) and encode object-level geometry.
• Cross-Category Challenges: Performance drops further in the Cross-SOC setting (e.g., DINOv2: 55.0%), highlighting limited abstraction capabilities across categories.
• Model Architectures: Dense self-supervised models (DINO family) outperform global alignment models (CLIP) and reconstruction-based models (MAE, CroCov2) on correspondence tasks. However, even the best models show significant gaps in geometric reasoning.
• Category Dependence: Performance varies significantly by super-category. Models struggle most with "Furniture" and "Transportation" due to repeated symmetric parts, while "Animals" show smaller CC→SOC gaps.

Large Vision-Language Models (LVLMs)

• Localization Gap: LVLMs are significantly stronger at text-prompted part localization (Desc. setting) than at visual-reference cross-image matching (Vis. setting). For instance, Qwen3-VL-8B achieves 54.0% accuracy in the Desc. setting but only 34.2% in the Vis. setting.
• Implication: This exposes a gap between language-grounded localization and fine-grained visual correspondence, suggesting LVLMs rely heavily on textual cues rather than robust visual feature alignment for part matching.

Relation to Downstream Tasks

• Stronger Predictor: SOC performance correlates more strongly with dense downstream tasks (segmentation, 3D detection, tracking, 3D pose) than ImageNet kNN classification.
• Diagnostic Value: This positions SOC as a practical, zero-shot diagnostic for representation quality that is more aligned with structured visual understanding than global classification metrics.

Significance and Claims

The paper positions SOCO as a unified benchmark for analyzing fine-grained visual reasoning and multimodal representation quality in the era of large foundation models. The authors claim that:

1. SOCO exposes specific failure modes: By separating CC, SOC, and Cross-SOC, the benchmark reveals that current models reliably match semantic concepts but struggle with object-level geometry and cross-category abstraction.
2. A new diagnostic standard: SOC serves as a more effective zero-shot diagnostic for representation quality than ImageNet classification, particularly for tasks requiring dense, structured understanding.
3. Multimodal limitations: The benchmark highlights a critical gap in current LVLMs, which are proficient at locating parts via text but lack the ability to transfer visual references across images with fine-grained precision.

The authors conclude that SOCO provides a stepping stone toward models that not only recognize objects but also understand their parts and structural relationships in a way that generalizes across categories and modalities. The dataset and code are made available to facilitate further research in this direction.