Are Object-Centric Representations Better At Compositional Generalization?

This paper introduces a comprehensive Visual Question Answering benchmark demonstrating that object-centric representations outperform dense vision encoders in compositional generalization, particularly under constraints of limited data diversity, smaller dataset sizes, or restricted downstream compute.

The Gist

Imagine you are teaching a child how to understand the world. You show them a red ball and a blue cube. Later, you show them a blue ball and a red cube.

A smart child (like a human) instantly understands: "Oh, I know what 'blue' is, and I know what a 'cube' is. I can mix and match these ideas to understand new things I've never seen before." This is called compositional generalization.

However, many current AI models are like a child who has only memorized specific flashcards. If you show them a "red ball," they know it. But if you show them a "blue cube" (a combination they haven't seen), they might get confused because they haven't memorized that specific card.

This paper asks a big question: Is there a better way to teach AI to think like a human? Specifically, does teaching AI to see the world as a collection of separate "objects" (Object-Centric) work better than teaching it to see the world as a giant, blurry picture (Dense)?

The Two Ways of Seeing

To explain the paper's findings, let's use two analogies:

1. The Dense Approach (The "Mosaic" or "Blurry Photo")
Imagine looking at a scene through a high-resolution camera that captures every single pixel. You see a red ball and a blue cube, but they are just a massive grid of colored dots.

• The Problem: To understand the "blue cube," the AI has to memorize the specific pattern of blue dots in that shape. If you change the shape to a sphere, the pattern of dots changes completely, and the AI struggles to connect the dots.
• The Fix: To make this work, you need to show the AI millions of pictures and use a massive brain (lots of computing power) to find the patterns.

2. The Object-Centric Approach (The "Lego Box")
Imagine looking at the same scene, but instead of pixels, you see a box of Legos. The AI doesn't see a "red ball"; it sees a Red Lego and a Round Lego snapped together. It sees a Blue Lego and a Square Lego.

• The Advantage: Because the AI has separated the "Red" from the "Round" and the "Blue" from the "Square," it can easily snap them together in new ways. If it sees a "Blue Round Lego," it knows exactly what that is, even if it's never seen that specific combination before.

The Experiment: A Visual Quiz Show

The researchers set up a giant quiz show to test these two approaches.

• The Setup: They created three different "worlds" (like video game levels) filled with objects of different shapes, colors, and sizes.
• The Trick: They trained the AI on some combinations (e.g., Red Cubes, Blue Spheres) but kept a secret "test" set of combinations the AI had never seen (e.g., Blue Cubes, Red Spheres).
• The Test: They asked the AI questions like, "Is the blue object a cube?" or "How many red things are there?"

They tested two types of AI "brains":

1. The Heavyweights: Giant, pre-trained models (like DINOv2 and SigLIP2) that see the world as a "Mosaic" (Dense).
2. The Organizers: Models built on top of the Heavyweights that force the AI to break the image down into "Legos" (Object-Centric).

The Results: Who Won?

The paper found that the answer depends on how much help the AI gets.

Scenario A: The "Hard Mode" (Limited Data or Computing Power)

• The Situation: You only have a few pictures to train the AI, or you don't have a supercomputer to run it.
• The Winner: The "Lego Box" (Object-Centric) approach wins easily.
• Why? Because it learns the rules of the world (Red + Cube = Red Cube) rather than memorizing the pictures. It's like teaching a child the alphabet; once they know the letters, they can read any word, even ones they haven't seen. The "Mosaic" approach gets lost in the details and fails to generalize.

Scenario B: The "Easy Mode" (Infinite Data and Supercomputers)

• The Situation: You have a billion pictures and a massive computer farm.
• The Winner: The "Mosaic" (Dense) approach can catch up and sometimes win.
• Why? If you show the "Mosaic" AI enough examples, it eventually memorizes every possible combination. It's like a child who has seen every single word in the dictionary; they can answer any question, but they needed to read the whole library to get there.

The Big Takeaway

The paper concludes that Object-Centric representations are the smarter, more efficient choice.

• Efficiency: They get better results with less data and less computing power.
• Robustness: They are better at handling "hard" new situations where the AI has to think creatively.
• The Catch: The "Mosaic" approach (the current standard in AI) only wins if you throw enough money and data at it to brute-force the solution.

In simple terms: If you want an AI that learns like a human—understanding concepts and mixing them together—you should teach it to see the world as separate objects (Legos), not just a giant pile of pixels. This is especially true if you don't have infinite resources to train it.

Technical Summary

1. Problem Statement

The paper addresses compositional generalization, the ability of a model to reason about novel combinations of familiar concepts (e.g., recognizing a "blue cube" when trained only on "red cubes" and "blue spheres"). While Object-Centric (OC) representations—which decompose scenes into discrete object slots—are theoretically hypothesized to support this capability better than dense, grid-based representations, systematic evidence in visually rich settings has been limited.

Previous studies often relied on simple synthetic worlds or generative tasks that did not isolate specific design choices. Furthermore, it remains unclear whether OC approaches outperform dense foundation models when accounting for fair comparisons in terms of representation size, downstream model capacity, and computational budget (FLOPs).

2. Methodology

A. Benchmark Creation

The authors introduce a controlled Visual Question Answering (VQA) benchmark spanning three synthetic visual worlds:

1. CLEVRTex: Rich textures and materials.
2. Super-CLEVR: Complex object shapes and attributes.
3. MOVi-C: Realistic 3D objects with diverse properties.

Dataset Construction:

• Object Definition: Objects are defined by combinations of factors (shape, size, material, color, etc.).
• Training/Testing Split: For each dataset, the full combinatorial space of object properties is generated.
  • Training Sets: Created by progressively restricting the diversity of objects seen during training (Easy: 80% of combinations, Medium: 40%, Hard: 20%).
  • Test Set (COOD): A fixed "Compositional Out-of-Distribution" set containing the remaining 20% of object property combinations that were never seen during training, though their individual properties were.
• VQA Generation: Question-answer pairs are generated for every image to evaluate reasoning capabilities.

B. Models and Experimental Setup

To ensure a fair comparison, the authors control for representation size and compute:

• Foundation Models:
  • Dense: DINOv2 (ViT-S/14) and SigLIP2 (ViT-B/16).
  • Object-Centric (OC): DINOSAURv2 (Slot Attention on DINOv2) and SigLIPSAUR2 (Slot Attention on SigLIP2).
• Fairness Controls:
  • Representation Size: Dense models produce many tokens (e.g., 256 or 196), while OC models produce few (e.g., 7 slots). The authors use a Cross-Attention (CA) layer to resize representations to match token counts, ensuring FLOPs are comparable.
  • Downstream Models: Two Transformer sizes (TF 2: 2 layers, TF 5: 5 layers) are used for the VQA task.
  • Baselines: Includes a "Question-Only" baseline and a "Ground-Truth Oracle" (using true object properties as input) to establish upper/lower bounds.

C. Evaluation Metrics

Performance is measured on:

1. In-Distribution (ID): Accuracy on test sets containing seen object combinations.
2. Compositional Out-of-Distribution (COOD): Accuracy on the held-out novel combinations.
3. Efficiency: Accuracy vs. Downstream Compute (FLOPs) and Sample Efficiency (accuracy vs. training data size).

3. Key Contributions

1. A Controlled Benchmark for Object-Property Composition: A VQA benchmark across three visually rich worlds (CLEVRTex, Super-CLEVR, MOVi-C) with explicit control over training diversity, enabling systematic evaluation of COOD generalization.
2. Fair Systematic Comparison: A rigorous evaluation framework that matches representation sizes, downstream model capacity, and FLOPs, isolating the effect of the representation structure (dense vs. object-centric) from computational advantages.
3. Empirical Evidence for OC Superiority: Demonstrating that OC representations are superior in constrained settings (limited data, diversity, or compute), while dense representations only catch up or surpass them in easy settings with abundant resources.

4. Key Results

Finding I: Training Diversity Controls Difficulty

As training diversity decreases (moving from Easy to Hard), ID accuracy increases (fewer combinations to learn), but COOD accuracy drops significantly. This widens the gap between ID and COOD performance, confirming the benchmark successfully isolates compositional generalization challenges.

Finding II: Representation Type Matters

• Hard Generalization: OC models (DINOSAURv2, SigLIPSAUR2) consistently outperform dense models (DINOv2, SigLIP2) in Hard COOD settings, especially with smaller downstream models (TF 2).
• Easy Generalization: Dense models can match or slightly surpass OC models in Easy settings, but this often requires significantly larger downstream models.
• Slot Attention vs. Alternatives: Slot Attention-based OC models outperform simple k-means clustering or cross-attention resizing methods, suggesting the "soft" binding of Slot Attention is crucial for preserving fine-grained visual details needed for VQA.

Finding III: Compute Efficiency

• Constrained Compute: Under limited FLOPs (small downstream models), OC models achieve higher COOD accuracy.
• Cost of Dense Models: To match the COOD performance of OC models on Hard tasks, dense models require substantially more compute (up to 3x). Even with increased compute, dense models often fail to surpass OC models on the hardest generalization tasks.

Finding IV: Sample Efficiency

• Data Scarcity: OC models are more sample-efficient. They achieve strong generalization with fewer training images.
• Data Abundance: Dense models only catch up to or surpass OC models when trained on the full dataset (40k images) with high diversity and a large downstream model.
• Diversity vs. Quantity: Increasing data diversity (seeing more unique object combinations) yields greater generalization gains than simply increasing the number of samples from a low-diversity set.

5. Significance and Conclusion

The paper provides strong evidence that Object-Centric representations are fundamentally better suited for compositional generalization when resources (data, diversity, or compute) are constrained.

• Theoretical Implication: The results support the hypothesis that decomposing scenes into discrete objects aligns better with the inductive biases required for systematic reasoning about novel combinations.
• Practical Implication: For applications requiring robust generalization to unseen scenarios (e.g., robotics, autonomous driving in novel environments) where collecting massive, diverse datasets is infeasible, OC approaches offer a more efficient and effective path than scaling up dense foundation models.
• Future Work: The authors suggest extending these findings to real-world scenarios beyond synthetic benchmarks and optimizing architectural choices to further enhance performance.

In summary, while dense foundation models are powerful, they rely heavily on massive data and compute to achieve compositional robustness. Object-centric learning offers a more sample-efficient and compute-efficient alternative for achieving systematic generalization.