Evaluating Object-Centric Models beyond Object Discovery

Technical Summary: Evaluating Object-Centric Models beyond Object Discovery

1. Problem Statement

Object-Centric Learning (OCL) aims to decompose scenes into latent object representations to enable compositional generalization, robustness to out-of-distribution (OOD) data, and causal reasoning. However, the authors identify two critical flaws in how these models are currently evaluated:

• Limited Scope of Evaluation: Most OCL models are evaluated using Unsupervised Object Discovery (UOD) (e.g., mIoU or mBO) or simple linear probing. The authors argue that UOD is a poor proxy for high-level reasoning, and linear probing is unscalable because it requires retraining a new probe for every specific task or capability (e.g., counterfactual reasoning), making it impossible to assess broad visual reasoning efficiently.
• Disjoint Evaluation Metrics: Current benchmarks evaluate localization ("where" an object is) and representation usefulness ("what" the object is) using separate, disconnected metrics. This leads to two failure modes:
  • Localization Fragmentation: A model captures the correct semantics but fails to localize the object accurately.
  • Representation Fragmentation: A single object's information is split across multiple latent "slots," which standard accuracy metrics fail to penalize.

2. Methodology

To address these issues, the paper proposes a two-pronged evaluation framework:

A. Scalable VLM-based Probing (Addressing Scope)
Instead of training task-specific probes, the authors use instruction-tuned Vision-Language Models (VLMs) as evaluators.

• Architecture: They replace the standard vision encoder (like CLIP or DINOv2) in a LLaVA-style architecture with an OCL encoder.
• Training: The process involves two stages: (1) training an MLP connector to align slot embeddings with the LLM's embedding space, and (2) instruction-tuning the LLM and connector on multimodal datasets.
• Benefit: This allows for zero-shot evaluation across diverse Visual Question Answering (VQA) benchmarks (e.g., GQA, POPE, MME), measuring how effectively a language model can exploit the object-centric slots to answer complex questions.

B. Attribution-Aware Grounded Accuracy (AwGA) (Addressing Disjoint Metrics)
To unify "what" and "where," the authors introduce a new metric and dataset:

• eGQA Dataset: An enhanced version of the GQA dataset where bounding boxes are converted into high-quality masks using SAM2.
• AwGA Metric: Unlike standard grounded accuracy (which only checks if the answer is correct and the mask overlaps), AwGA uses gradient-based attribution to identify which specific slots the VLM used to generate the answer. It then calculates the mIoU of the top-K most influential slots against the ground-truth mask. This directly penalizes both localization errors and representation fragmentation.

3. Key Contributions

1. A Scalable Evaluation Framework: A VLM-based protocol that enables zero-shot benchmarking of OCL models across diverse reasoning tasks without retraining.
2. AwGA Metric: A novel, unified metric that jointly assesses localization and representation usefulness by incorporating attribution.
3. Empirical Analysis of OCL Utility: A comprehensive study showing that object discovery scores do not correlate with downstream reasoning utility.
4. mFRESA Baseline: A multi-feature reconstruction baseline (combining image, DINOv2, and HOG features) that demonstrates how to improve OCL performance.

4. Key Results

• Utility vs. Discovery: The authors found that models with high object discovery scores (like FT-DINOSAUR) often perform worse on VQA and robustness tasks than models with lower discovery scores. This proves UOD is an unreliable metric for reasoning capability.
• Performance Benchmarks: While OCL models are competitive with DINOv2 in terms of token efficiency (using only 7 slots vs. 196 patches) and show strength in OOD generalization and numeric counterfactual reasoning, they still lag behind foundational models in compositional reasoning and natural adversarial robustness.
• Effectiveness of mFRESA: The proposed multi-target reconstruction baseline (mFRESA) consistently outperformed existing OCL methods across most benchmarks, suggesting that combining different reconstruction signals (pixels + features) is vital for useful representations.
• Metric Robustness: The AwGA metric proved robust to different LLM backbones (Phi-2 vs. Qwen2) and different attribution methods (Gradients vs. Integrated Gradients).

5. Significance

This paper shifts the paradigm of OCL evaluation from "can the model find objects?" to "can the model use objects to reason?" By providing a scalable, unified, and attribution-aware evaluation suite, the authors provide the community with the tools necessary to develop next-generation object-centric models that are truly useful for complex, multimodal, and compositional AI tasks.