When Slots Compete: Slot Merging in Object-Centric Learning

1. Problem Statement

Object-Centric Learning (OCL) aims to decompose visual scenes into discrete, meaningful entities (objects) using unsupervised methods. The dominant framework, Slot Attention, represents a scene as a fixed set of $K$ latent vectors ("slots") that iteratively compete to explain different regions of an image.

The core limitation addressed in this paper is Slot Competition (or fragmentation):

• Fixed Cardinality: Slot Attention requires a pre-defined number of slots $K$. In real-world scenes with variable object counts, setting $K$ higher than necessary leads to over-segmentation.
• Redundancy: Instead of discarding redundant slots, multiple slots often compete to attend to the same object. This fragments the representation of a single entity across multiple slots, wasting representational capacity and degrading segmentation quality.
• Limitations of Existing Solutions: Previous adaptive methods (e.g., Adaslot, MetaSlot) address over-segmentation by pruning or suppressing redundant slots. The authors argue that discarding slots is suboptimal; if multiple slots represent the same object, their information should be merged into a single coherent representation rather than deleted.

2. Methodology

The authors propose a Slot Merging mechanism that acts as a lightweight, differentiable refinement step integrated into the standard Slot Attention pipeline (specifically within the DINOSAUR framework).

A. Quantifying Spatial Competition (Soft-IoU)

To identify which slots represent the same object, the method analyzes the attention maps generated by Slot Attention.

• Let $A$ be the normalized attention matrix where $A_{n,k}$ is the attention weight of slot $k$ on spatial location $n$.
• The overlap between two slots $i$ and $j$ is quantified using a Soft Intersection-over-Union (Soft-IoU) score:
  $$\text{IoU}(p, q) = \frac{\sum_n p_n q_n}{\sum_n (p_n + q_n - p_n q_n)}$$
  where $p$ and $q$ are the attention vectors for slots $i$ and $j$. A high IoU indicates significant spatial overlap, suggesting competition for the same entity.

B. Differentiable Slot Merge Operator

Once a pair of competing slots $(i, j)$ is identified, they are merged using a barycentric update that preserves gradient flow:

1. Mass Calculation: The "mass" (total attention weight) of each slot is calculated: $\alpha_i = \sum_n A_{n,i}$.
2. Weighted Interpolation: The new merged slot representation $S_i$ is a convex combination of the original slots, weighted by their relative mass:
   $$S_i \leftarrow w_i S_i + w_j S_j, \quad \text{where } w_i = \frac{\alpha_i}{\alpha_i + \alpha_j}$$
3. Attention Map Aggregation: The attention maps are summed to preserve the total spatial support:
   $$A_{:,i} \leftarrow A_{:,i} + A_{:,j}$$
4. Removal: Slot $j$ is removed from the set.

• Key Feature: This operation is differentiable, allowing gradients to propagate back through the merge operation to the original slot representations during training.

C. Fixed Merge Policy & Stopping Rule

The merging process is governed by a fixed policy:

• Selection: The pair with the maximum IoU ($I_{max}$) is selected for merging.
• Termination: The process repeats iteratively until the maximum pairwise overlap falls below a threshold $\tau$ ($I_{max} < \tau$).
• Threshold Estimation: The threshold $\tau$ is not learned but inferred from the empirical distribution of overlap scores using triangle thresholding on a histogram of pairwise overlaps.
• Training Schedule: Merging is disabled during the initial training phase (to allow slots to form) and activated only after slot representations have stabilized (e.g., after epoch 70 on VOC).

3. Key Contributions

1. Spatial Slot Competition Score: Introduction of a probabilistic Soft-IoU metric to detect overlapping slot attention maps, identifying fragmented representations of the same object.
2. Differentiable Merge Operator: A novel operator that merges competing slots via mass-weighted convex interpolation. Unlike pruning methods, this preserves information and allows gradient flow, optimizing the merged representation jointly with the network.
3. Fixed Merge Policy: A data-driven, non-learnable policy that selects merge candidates based on overlap statistics and regulates the process with an adaptive stopping threshold, requiring no additional learnable parameters.
4. Empirical Validation: Comprehensive evaluation showing that merging improves object factorization and segmentation quality across real-world (VOC, COCO) and synthetic (MOVi) benchmarks.

4. Experimental Results

The method was integrated into DINOSAUR (a state-of-the-art feature-reconstruction OCL framework) and evaluated on four datasets: PASCAL VOC 2012, MS COCO 2017, MOVi-C, and MOVi-E.

• Performance Gains:
  • On VOC, the method achieved 55.34 mBOc (Mean Best Overlap class), outperforming the baseline DINOSAUR (51.2) by over 4 points.
  • On COCO, it surpassed strong baselines including DINOSAUR+FS+RC, achieving 42.78 mBOc.
  • On synthetic datasets (MOVi), it showed consistent improvements in both mBO and mIoU.
• Ablation Studies:
  • Training vs. Inference: Integrating merging during training yielded significantly better results than applying it only at inference, proving that the network learns better representations when aware of the merging process.
  • Differentiability: Detaching gradients during the merge step degraded performance, confirming that the improvement relies on the joint optimization of slot representations and the merge operation.
  • Attention Update: Aggregating attention maps (Eq. 5) was shown to be crucial; retaining only one map led to performance drops.
• Comparison: The method outperformed other adaptive slot cardinality approaches like Adaslot and MetaSlot, demonstrating that differentiable aggregation is superior to discrete pruning or clustering for handling slot redundancy.

5. Significance

This paper reframes the problem of over-segmentation in Object-Centric Learning. Instead of viewing redundant slots as noise to be discarded, the authors treat them as fragmented representations that should be consolidated.

• Paradigm Shift: It moves from "slot selection/suppression" (discrete, hard decisions) to "slot consolidation" (continuous, differentiable refinement).
• Efficiency: The method is lightweight, adding minimal computational overhead ($O(NK^2)$) compared to the Slot Attention updates, and requires no new learnable modules.
• Generalizability: By improving the quality of object-level representations, the method enhances downstream tasks such as object discovery, segmentation, and reasoning, making OCL more robust for complex, real-world scenes with variable object counts.