Technical Summary: DeCoDrift: Stabilizing Decoder Coupling in Closed-Loop Foundation Segmentation

Problem Statement

Foundation segmentation models, such as the Segment Anything Model (SAM), are increasingly deployed in iterative pipelines where predicted masks from one iteration serve as prompts for the next. This creates a closed-loop dynamical system. While prior research has focused on input-side robustness (e.g., prompt sensitivity, corruption robustness), the internal behavior of the decoder during these iterative feedback loops remains unexamined.

The paper identifies a previously overlooked failure mode termed decoder coupling drift. In this phenomenon, the mask decoder's cross-attention progressively loses alignment with the target object across iterations. This misalignment causes errors to compound: small inaccuracies in a predicted mask ($M_t$) generate shifted prompts ($p_{t+1}$), which alter the decoder's cross-attention, leading to further degradation in the subsequent mask. The central question addressed is: What happens inside SAM's decoder during this iterative drift, and can we intervene at inference time without retraining?

Methodology

1. Instrumentation and Metrics

The authors instrumented SAM's two-layer mask decoder at four specific attention sites: final self-attention, token-to-image cross-attention, image-to-token cross-attention, and final token-to-image attention. Based on these internal signals, they derived ten metrics to quantify the coupling between decoder attention and the target without relying on ground truth for monitoring:

• Ground-Truth-Free Signals: Attention Anchor Drift (AAD), Temporal Coupling Stability (TCS), Prompt Drift Energy (PDE), Attention Entropy (AE), and Self-Cross Correlation (SCA).
• Ground-Truth-Dependent (for analysis): Cross-Attention Mask Alignment (CAMA), Distractibility Leakage Ratio (DLR), Causal Coupling Drop (CCD), and a composite Prompt-Image Coupling Score (PICS).

2. Dynamical Systems Analysis

The iterative prompting process is formalized as a discrete-time dynamical system: $p_{t+1} = \Phi_\theta(I_t, p_t)$.

• Stability Analysis: The authors prove that the feedback loop is asymptotically stable only if the spectral radius of the Jacobian, $\rho(J_t)$, is less than 1. When $\rho(J_t) \geq 1$, errors grow geometrically, manifesting as the observed drift.
• Proximal Stabilization: To counteract this, they propose replacing the standard update with a regularized update derived from proximal operators. This update blends the current prediction with an initial anchor ($p_0$) and the previous prompt ($p_{t-1}$), effectively reducing the spectral radius of the system and bounding error amplification.

3. DeCoDrift Framework

Building on the analysis, the authors introduce DeCoDrift, a training-free, inference-time stabilization framework comprising four components:

1. Attention-Guided Prompt Extraction: Instead of using raw mask moments, prompts (centroids/bboxes) are extracted using the decoder's own attention map as a spatial prior to bias extraction toward attended regions.
2. Proximal Anchor Regularization: The extracted prompt is regularized using the derived closed-form solution, blending the new prompt with a fixed anchor and the previous prompt to prevent unbounded drift and oscillation.
3. Decoder-Aware Candidate Selection: When multiple candidate masks are generated, the selection criteria are modified to trade off confidence, continuity, uncertainty, divergence, and leakage, rather than simply maximizing IoU.
4. Soft Object Persistence: A mechanism to retain objects that are temporarily occluded or unstable, using AAD/TCS thresholds and anchor-based recovery, replacing hard filtering.

Key Contributions

1. Decoder Coupling Framework: The first systematic instrumentation of a foundation model's decoder to define metrics quantifying attention-target coupling without ground truth.
2. Empirical Characterization: Demonstration on volumetric electron microscopy (MitoEM) data that standard iterative prompting suffers from systematic attention drift, elevated prompt energy, and degraded temporal stability compared to an oracle-anchored upper bound.
3. Dynamical Analysis: Formalization of the prompt-feedback loop as a dynamical system, proving that proximal anchor regularization bounds the effective Jacobian and stabilizes the system.
4. Training-Free Stabilization: The introduction of DeCoDrift, which improves segmentation quality and stability without retraining, weight modification, or additional data.

Experimental Results

Experiments were conducted on the MitoEM dataset (rat cortex mitochondria) using 10 volumetric slices and 133,452 paired decoder calls.

• Coupling Metrics: Standard iterative prompting (Method A) exhibited significantly higher Attention Anchor Drift (AAD) and lower Temporal Coupling Stability (TCS) compared to the oracle (Method B). Specifically, TCS dropped from 0.995 (oracle) to 0.847 (iterative), indicating a 15% shift in attention per step.
• Stabilization Performance: DeCoDrift (A-Stab) closed 76.4% of the TCS gap and 27.9% of the AAD gap relative to the oracle.
• Segmentation Quality: DeCoDrift improved the end-to-end Intersection over Union (IoU) by 37.7% (relative) compared to the standard iterative baseline at short distances (≤10 slices).
• Ablation: The proximal regularization component was identified as the dominant contributor to stability, reducing AAD by 18% and increasing TCS by 12.2 percentage points on its own.
• Runtime: The framework adds approximately 4.1 ms (4.9%) overhead per object per iteration on an A5000 GPU.

Significance and Claims

The paper claims that decoder-internal dynamics are not merely diagnostic but provide actionable signals for stabilizing foundation models in closed-loop use. The key findings are:

• Decoupling is Systematic: Iterative prompting induces a systematic failure mode where attention drifts away from the target, invisible from the output mask alone until errors accumulate.
• Internal Signals Predict Quality: Metrics like AAD and TCS are strongly predictive of segmentation quality (Pearson/Spearman correlations $|r| > 0.95$) when coupling is controlled.
• Stability vs. Adaptability: DeCoDrift achieves a "sweet spot" by bounding drift through proximal regularization while preserving enough flexibility for slice-to-slice adaptation, without requiring ground truth anchors during inference.
• Generalizability: The authors suggest that any system iteratively querying a foundation model (e.g., chain-of-thought prompting, iterative diffusion) may exhibit analogous drift, and their framework is architecture-agnostic.

The authors remain modest regarding limitations, noting that evaluation is currently restricted to a single primary dataset (MitoEM) and that the oracle-anchored method serves only as an upper bound. They acknowledge that while DeCoDrift stabilizes the dynamic part of the loop, static coupling (causal dependence) is harder to alter without weight modification.