Through the Looking Glass: A Dual Perspective on Weakly-Supervised Few-Shot Segmentation

Technical Summary: Through the Looking Glass (TLG)

Problem Statement
Current meta-learning approaches for few-shot semantic segmentation typically employ identical network architectures for both support and query branches. While these methods aim to extract inductive biases from homologous support-query pairs (images sharing the same category), the authors argue that this "homologous network" design leads to over-semantic homogenization. By forcing both branches to operate within the same semantic mapping space, models tend to overemphasize shared features (e.g., a "head" common to all dogs) while neglecting instance-level heterogeneous details (e.g., specific coat colors or sizes). This results in under-segmentation and limits the model's ability to capture complementary semantic information. Furthermore, in the context of Weakly-Supervised Few-Shot Semantic Segmentation (WFSS), where only image-level labels are available (lacking precise pixel-level masks), existing state-of-the-art (SOTA) models often suffer from high parameter counts and computational costs, yet still fail to fully exploit the potential of multimodal priors.

Methodology
The paper proposes TLG (Through the Looking Glass), a novel homologous but heterogeneous network designed to treat support-query pairs as dual perspectives. The framework is built upon three core modules:

1. Heterogeneous Aggregation (HA): Instead of extracting features from identical backbone layers for both branches, TLG employs a differentiated layer selection strategy. For instance, the support branch may extract features from layers 3, 9, and 12, while the query branch extracts from layers 0, 4, and 10. This intra-layer heterogeneity maximizes semantic richness by leveraging low-level details (query) and high-level semantics (support) simultaneously, while preserving semantic commonality through concatenation and correlation mechanisms.
2. Heterogeneous Transport (HT): To address the semantic noise inevitably introduced by aggregating heterogeneous features, the HT module utilizes Optimal Transport (OT). It models the alignment of heterogeneous semantic information as a transport cost minimization problem (solved via the Sinkhorn algorithm). This process suppresses noise and redundant features while enhancing contextual relevance through cross-attention mechanisms. Additionally, Heterogeneous Residuals (HR) are introduced to preserve a portion of the original input features, ensuring representation diversity is not lost during alignment.
3. Heterogeneous CLIP (HC): To enhance generalization, TLG integrates multimodal textual information from CLIP. Unlike previous methods that map all background priors to a common space, TLG introduces a maximum matching strategy. It uses a Chain-of-Thought approach (via LLMs like ChatGPT) to identify specific background categories co-occurring with foreground objects (e.g., "bird" with "tree" and "sky"). These fine-grained, heterogeneous text prompts (e.g., "bird with feathers") are fused with visual features via a heterogeneous CLIP adapter, improving the alignment between textual priors and visual semantics.

Key Contributions

• Paradigm Shift: The authors identify the limitation of homogeneity in meta-learning and propose a homologous but heterogeneous network design, arguing that distinct but semantically aligned architectures better capture the "heterogeneity of the heterogeneous."
• Novel Modules: The introduction of the HA, HT, and HC modules to specifically address semantic over-homogenization, noise reduction in heterogeneous aggregation, and multimodal generalization, respectively.
• Efficiency and Performance: TLG achieves significant performance gains with drastically reduced computational resources. It utilizes only 1/24 of the trainable parameters of existing SOTA models (specifically AFANet).
• Weakly-Supervised Breakthrough: The paper claims TLG is the first weakly-supervised (image-level label) model to outperform fully-supervised (pixel-level label) models under the same backbone architectures.

Experimental Results
Extensive experiments were conducted on the Pascal-5i and COCO-20i datasets using image-level labels.

• Pascal-5i: With a ResNet50 backbone, TLG achieved a 77.2 mIoU in the 1-shot setting and 77.9 mIoU in the 5-shot setting. This represents a 13.2% improvement over the SOTA weakly-supervised model AFANet.
• COCO-20i: TLG achieved a 51.5 mIoU (1-shot) and 51.6 mIoU (5-shot), showing a 7.9% improvement over AFANet.
• Lightweighting: Despite the performance gains, TLG requires only 4.47M learnable parameters (compared to AFANet's 108.35M) and significantly lower FLOPs and inference latency (e.g., 36.72ms vs. 73.21ms for 1-shot inference on ResNet50).
• Comparison with Fully-Supervised: TLG outperforms fully-supervised SOTA models (e.g., VRP-SAM, PGMA-Net) on Pascal-5i, despite using only coarse image-level labels.

Significance and Claims
The paper posits that the core of effective meta-learning lies not in forcing uniformity, but in leveraging complementary heterogeneity. By treating support and query as dual perspectives with distinct feature extraction paths, TLG unlocks the full potential of meta-learning models. The authors claim that TLG sets a new benchmark for weakly-supervised learning, demonstrating that image-level supervision, when combined with a heterogeneous network design and multimodal priors, can surpass the performance of fully-supervised counterparts. This suggests a shift in design philosophy where "heterogeneity" is not just a challenge to be overcome, but a resource to be engineered for better generalization in data-scarce scenarios.