DC-TTA: Divide-and-Conquer Framework for Test-Time Adaptation of Interactive Segmentation

1. Problem Statement

Interactive Segmentation (IS) allows users to refine object boundaries using minimal cues (positive/negative clicks). While the Segment Anything Model (SAM) has shown strong zero-shot capabilities, it struggles in complex scenarios such as:

• Camouflaged objects or objects with complex boundaries.
• Multi-part objects where different parts require distinct attention.
• Conflicting user cues: When a single model attempts to incorporate all user clicks simultaneously (monolithic approach), conflicting information (e.g., a click on a background region vs. a foreground region) can cause the model to fail to update correctly, leading to false positives or suboptimal masks.

Existing Test-Time Adaptation (TTA) methods for IS typically update a single model using all accumulated clicks at once. This "all-at-once" approach often overwhelms the model, preventing it from effectively learning from diverse or conflicting sequential inputs.

2. Methodology: DC-TTA

The authors propose DC-TTA, a novel framework based on a Divide-and-Conquer (DC) strategy. Instead of updating one monolithic model, DC-TTA partitions user interactions into coherent subsets, adapts specialized models for each subset, and merges them.

Core Components:

1. Segmentation Units (Divide):

   • The framework maintains a set of Segmentation Units ($U^k$). Each unit encapsulates:
     • A subset of positive clicks ($P^k$).
     • A predicted mask ($M^k$).
     • Adapted model parameters ($\theta^k$).
   • A Global Unit ($U^0$) is maintained to capture the overall context by encompassing all clicks.
   • Negative clicks ($N$) are treated as auxiliary constraints applied uniformly to all units.

2. Click Assignment Strategy:

   • When a new positive click arrives, the system generates a temporary mask ($Q_t$) using only that click and existing negative clicks.
   • It calculates the Intersection over Union (IoU) between $Q_t$ and the masks of existing units.
   • Assignment: If IoU > 0, the click is assigned to the existing unit. If no overlap exists, a new unit is created specifically for that click. This ensures that clicks corresponding to distinct object parts or regions are handled by specialized models.

3. Local Test-Time Adaptation (Conquer):

   • Each unit performs TTA independently.
   • The unit uses the new click to generate an intermediate mask (pseudo-ground truth) and updates its specific parameters ($\theta^k$) via gradient descent to minimize the loss between the previous prediction and the new intermediate mask.
   • This allows the model to focus on localized details without being confused by conflicting cues from other parts of the image.

4. Model Merging (Integration):

   • Pixel-level Union: The masks from all units are aggregated via a pixel-wise union to form an initial unified mask.
   • Parameter-level Merging: To leverage the specialized knowledge encoded in the adapted parameters, the framework uses Task Vectors ($\tau^k = \theta^k - \theta_0$).
   • The final model parameters are computed by adding the scaled task vectors of all units to the original pre-trained parameters:
     $$\theta_{merged} = \theta_0 + \gamma\tau^0 + \gamma^2 \sum_{k=1}^{K} \tau^k$$
   • A final fine-tuning step is performed using the aggregated pixel mask as pseudo-ground truth to stabilize the merged model.

3. Key Contributions

• Divide-and-Conquer Strategy for IS: The paper introduces a paradigm shift from monolithic adaptation to partitioning user clicks into coherent subsets, allowing for specialized, conflict-free model updates.
• Novel TTA Framework (DC-TTA): A complete pipeline that dynamically creates segmentation units, performs independent TTA, and merges results via task vectors.
• Task Vector Merging: The application of task arithmetic to merge locally adapted models, effectively consolidating specialized knowledge into a unified predictor.
• Generalizability: The method is shown to improve not only SAM but also conventional IS methods (e.g., RITM, FocalClick, MFP), demonstrating broad applicability.

4. Experimental Results

The method was evaluated on eight IS benchmarks, including challenging datasets like CAMO (camouflaged objects), COD10K, and TRASHCAN (underwater trash).

• Performance Metrics: Evaluated using Number of Clicks (NoC) to reach 85%/90% IoU and Failure Rate (FR).
• Comparisons: DC-TTA consistently outperformed:
  • Baseline SAM (Zero-shot).
  • TENT (Standard TTA).
  • AdaptSAM (Previous IS-specific TTA).
  • DC-only (Divide-and-Conquer without TTA), proving that both the partitioning strategy and the adaptation process contribute to performance.
• Key Findings:
  • Significant reduction in NoC and FR, particularly in complex scenarios like camouflaged objects where baseline methods fail.
  • Ablation studies confirmed that the combination of DC strategy, TTA, and Model Merging yields the best results, with each component providing non-overlapping improvements.
  • Integration with non-SAM IS methods also yielded performance gains, validating the framework's versatility.

5. Significance

DC-TTA addresses a critical limitation in current interactive segmentation: the inability of a single model to simultaneously process diverse and potentially conflicting user inputs. By decomposing the problem into localized, coherent sub-problems and re-integrating them via parameter merging, the framework achieves:

• Higher Accuracy: Better handling of complex boundaries and multi-part objects.
• Efficiency: Fewer user interactions (clicks) are required to achieve high-quality segmentation.
• Robustness: Reduced susceptibility to false positives caused by conflicting cues.

This work establishes a new direction for Test-Time Adaptation in interactive tasks, suggesting that "specialized experts" (local units) combined with a "global coordinator" (merging) is a superior strategy for handling dynamic, user-driven inputs compared to monolithic updates.