Technical Summary: Learning to Label (L2L)

Problem Statement

Semi-supervised Referring Expression Segmentation (SS-RES) aims to achieve precise pixel-level language grounding using a small set of labeled image-text pairs and a large pool of unlabeled data. While existing semi-supervised segmentation methods rely on pseudo-labeling and consistency regularization, directly applying these paradigms to SS-RES is challenging due to referential ambiguity, occlusion, and the need for long-range contextual reasoning. These factors often lead to boundary uncertainty and unstable pseudo-masks.

Current approaches typically filter pseudo-labels using a fixed confidence threshold. However, this heuristic is brittle: overly strict thresholds discard informative samples, while loose thresholds introduce noisy supervision that contaminates training. Furthermore, recent works leverage Multimodal Large Language Models (MLLMs) and foundation models (e.g., SAM) to generate external priors. Yet, these priors are not always reliable; MLLMs may hallucinate or suffer from grounding drift, and segmentation models may over-segment or be distracted. Crucially, there is often a significant, sample-dependent disagreement between the confidence of the MLLM prior and the segmentation model's own predictions, rendering fixed-threshold strategies ineffective.

Methodology

The authors propose Learning to Label (L2L), a reinforced self-evolving framework that reframes pseudo-label construction as a learnable decision-making process. The framework operates in a closed loop where multimodal priors, pseudo-label selection, and the segmentation network co-evolve. The architecture consists of three core components:

1. Semantic-Spatial Prior with MLLM (SPM)

To initialize soft segmentation proposals, L2L leverages a parameter-frozen MLLM to extract semantic grounding cues for unlabeled image-text pairs. These cues prompt SAM2 to generate dense, soft segmentation proposals ($P^\dagger_i$) while preserving uncertainty.

• Uncertainty-Aware Calibration: Recognizing that external priors and model predictions ($P^w_i$) may conflict, SPM fuses them in logit space using a pixel-adaptive weight $\lambda(u)$. This weight is computed based on:
  • Pixel-wise certainty: A margin-based score $C(p)$ that increases as predictions move away from ambiguity (0.5).
  • Sample-level agreement: A global consistency score $W$ measuring the agreement between the weak prediction and the external prior.
• Fusion: The calibrated prior $\tilde{P}^\dagger_i$ is generated by weighting the logits of the external prior and the model prediction. This ensures that the model relies more on the external prior when it is uncertain, but suppresses the prior when the model is confident or when global agreement is low.

2. Self-Evolving Segmentation Module (SESM)

SESM integrates the calibrated prior into a hierarchical segmentation network. Instead of static fusion, it employs a stage-adaptive gating mechanism to inject semantic and structural guidance.

• Dual Guidance: The module combines a semantic field ($S$) from the text encoder and a structural field ($G_j$) derived from the pixel-wise prior.
• Dynamic Gating: A unified conditioning signal $M_j$ is synthesized via learnable stage-wise gating factors ($\alpha_j, \beta_j$) that balance high-level semantic disambiguation and low-level structural refinement.
• Stage-Dependent Interaction: For shallow stages, the structural field is applied via strict local projection to preserve spatial correspondence. For deep stages, global relational reasoning is enabled via cross-attention.
• Self-Evolution: The structural guidance is iteratively updated as the model's weak predictions improve, progressively reducing noise and increasing reliability.

3. Reinforced Pseudo-Label Exploration (RPLE)

To address the suboptimality of fixed thresholds, RPLE formulates pseudo-label acceptance as a reinforcement learning (RL) decision process.

• State Representation: The state $s_t$ includes statistical features of the fused prediction (mean, std), the proportions of foreground/background/ignored pixels, and the sample-level agreement score.
• Policy Searcher: An Actor network ($\pi_\phi$) outputs adaptive foreground and background confidence thresholds ($\tau_{fg}, \tau_{bg}$) for each sample.
• Critic & Reward: A Critic network ($Q_\phi$) evaluates the selected pseudo-labels. The reward is defined as an Instructional Gain ($\Delta\Phi_t$) comprising three components:
  • Separability Gain ($\Delta M$): Measures the contrastive boundary between foreground and background.
  • Consistency Gain ($\Delta K$): Measures the agreement between the model prediction and the calibrated prior.
  • Coverage Gain ($\Delta C$): Measures the fraction of selected pixels.
  • A stability penalty is applied to prevent erratic threshold changes.
• Optimization: The policy is optimized via deterministic policy gradients to maximize the expected cumulative quality score, dynamically balancing precision and coverage throughout training.

Key Contributions

1. Learning to Label Framework: The authors introduce L2L, which transforms pseudo-label construction from a static thresholding process into a learnable, sample-adaptive decision-making process, enabling more reliable mining of pixel-level supervision under sparse annotations.
2. MLLM-Driven Priors and Calibration: The work introduces MLLM-driven semantic-spatial priors to initialize soft segmentation and proposes a reinforced selection mechanism that adaptively identifies high-utility supervision by fusing multimodal priors with model predictions.
3. Empirical Validation: Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg demonstrate consistent improvements over existing semi-supervised and baseline methods across various label budgets (0.1% to 10%).

Experimental Results

• Performance: L2L achieves state-of-the-art results across all datasets and label budgets. Notably, under the extremely low-label regime (0.1% labeled data), L2L outperforms zero-shot counterparts and existing semi-supervised baselines (e.g., RESMatch, SemiRES).
• Ablation Studies:
  • Removing SPM, SESM, or RPLE individually leads to performance degradation, confirming the necessity of uncertainty-aware fusion, hierarchical guidance, and adaptive thresholding.
  • The weighting strategy in SESM (dynamic gating) outperforms fixed-weight fusion, validating the need to balance semantic and structural cues dynamically.
  • RPLE significantly outperforms static thresholding, which is shown to be highly sensitive to the choice of threshold values.
• Robustness: The method is robust to variations in foundation model choices (e.g., SAM vs. SAM2, different MLLM backbones) and remains effective even when MLLM priors are replaced by non-MLLM teachers (e.g., CARIS, CGFormer).
• Efficiency: The additional trainable parameters introduced by L2L are modest (~9.3M), and the RL components are lightweight, adding negligible inference-time cost as the foundation models remain frozen.

Significance and Claims

The paper claims that L2L provides a robust solution for building data-efficient referring segmentation systems under extremely low annotation costs. By treating pseudo-labeling as a reinforced, self-evolving process, the framework effectively mitigates the brittleness of fixed-threshold heuristics and the unreliability of external priors. The authors assert that their approach enables the joint optimization of the segmentation model and pseudo-labels, progressively enhancing label reliability. The work is presented as a step toward generalizable SS-RES systems that can leverage the reasoning capabilities of MLLMs without being hindered by their potential hallucinations or grounding drifts.