💻 computer science

Semi-Supervised Adaptation of Vision-Language Models for Image Classification

This paper introduces SE-CLIP, a semi-supervised framework that enhances Vision-Language Models for remote sensing image classification by employing a dual-phase pipeline with class-balanced recursive label mining to overcome annotation scarcity and outperform existing methods on UCM and NWPU benchmarks.

Original authors: Mohamed L. Mekhalfi, Mohamad M. Al Rahhal, Yakoub Bazi, Salah E. Khenfer, Mingdeng Shi, Hua Zou, Mansour Zuair

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mohamed L. Mekhalfi, Mohamad M. Al Rahhal, Yakoub Bazi, Salah E. Khenfer, Mingdeng Shi, Hua Zou, Mansour Zuair

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of modern artificial intelligence, a new generation of computer vision systems has emerged, trained not just to see images but to understand the words that describe them. These systems, known as vision-language models, learn by studying millions of pairs of pictures and text, building a shared mental map where visual concepts and language concepts sit side by side. This allows them to recognize objects in new situations without needing specific training for each one, a capability often called zero-shot learning. However, while these models excel at understanding everyday scenes like parks or city streets, they often stumble when faced with the unique, top-down perspective of satellite imagery. The colors, textures, and patterns of the Earth as seen from space are fundamentally different from the ground-level photos these models usually study. To make these powerful tools work for remote sensing, scientists typically need to fine-tune them with new data, but labeling thousands of satellite images by hand is slow, expensive, and requires specialized geographical knowledge. This creates a bottleneck where the technology exists, but the human effort required to teach it the specifics of our planet is too great.

To bridge this gap, a team of researchers has developed a new method called SE-CLIP, which allows these advanced models to teach themselves how to classify satellite images using very few human-labeled examples. Instead of relying on a massive dataset of pre-labeled images, the system starts with a tiny handful of expert-verified samples, just five images for each type of land cover, such as airports, forests, or residential areas. From this small starting point, the framework enters a recursive cycle of discovery. It uses the model's existing understanding of language to generate text descriptions for each category, effectively creating a set of textual anchors. The system then scans a vast pool of unlabeled satellite images, searching for those that visually match these text descriptions with high confidence. Crucially, the researchers designed the system to be fair: it selects an equal number of high-confidence images for every single category, preventing the model from becoming obsessed with the easiest or most common types of land cover while ignoring the complex or rare ones. These newly identified images are then added to the training set, and the process repeats, allowing the model to continuously evolve and refine its understanding of the Earth's surface.

The results of this approach are striking, particularly when tested on two major benchmarks for satellite image classification: the UC Merced dataset, which contains 2,100 images across 21 categories, and the much larger NWPU-RESISC45 dataset, which holds 31,500 images across 45 categories. In the initial phase, the model warms up using only the few labeled seeds to stabilize its learning. Once this foundation is set, the recursive discovery phase begins. On the smaller UC Merced dataset, the system achieved a peak accuracy of 99.09% by utilizing a LoRA rank of 16. On the more complex and challenging NWPU dataset, the system performed even better as it was allowed to mine more samples, reaching a peak accuracy of 95.07% with the same configuration. The researchers found that if they tried to mine too many images at once, the accuracy dipped slightly, suggesting that including too many lower-confidence samples introduced noise into the learning process. Furthermore, the study demonstrated that this method is highly efficient, requiring the training of only a tiny fraction of the model's total parameters, which keeps the computational cost low while delivering results that significantly outperform existing semi-supervised techniques.

A key finding of the research is that the method's success depends heavily on how it manages the balance between different categories. When the researchers tested a version of the system that did not enforce this balance, simply picking the best images regardless of their type, the performance collapsed. On the NWPU dataset, the unbalanced approach dropped to an accuracy of just 42.33%, whereas the balanced approach maintained an accuracy of 94.92%. This dramatic difference highlights that without a strict rule to ensure every land-use category gets an equal share of new training data, the model tends to ignore difficult or less frequent classes, leading to a skewed and unreliable understanding of the landscape. Visualizations of the model's internal representation of the data confirm this: while the original, untrained model sees all categories as a jumbled, overlapping mess, the SE-CLIP method organizes these images into tight, distinct clusters, clearly separating one type of land cover from another.

The study also compared this new framework against several other state-of-the-art methods designed for semi-supervised learning. In every case, SE-CLIP came out on top. On the UC Merced dataset, it achieved 99.09% accuracy, surpassing the next best method by more than three percentage points. On the more difficult NWPU benchmark, it reached 95.07%, outperforming the strongest competitor by over six percentage points. These results suggest that combining the broad, pre-trained knowledge of a vision-language model with a careful, balanced strategy for self-learning offers a powerful solution for remote sensing. The framework effectively turns a small amount of human expertise into a large, high-quality dataset without the need for continuous manual labeling. While the system relies on the ability to describe land types with words, and may struggle with extremely subtle differences between similar-looking categories, it provides a viable path forward for adapting these powerful AI tools to the specialized needs of monitoring our planet from space.

Technical Summary: Self-Evolutionary CLIP (SE-CLIP) for Remote Sensing

Problem Statement
Vision-Language Models (VLMs), particularly Contrastive Language-Image Pre-training (CLIP), have demonstrated significant potential for zero-shot classification in Remote Sensing (RS) by establishing a unified latent space. However, a critical performance gap exists due to the domain discrepancy between general-purpose natural image pre-training and the specific spectral-spatial geometries of satellite imagery. While parameter-efficient fine-tuning techniques like Low-Rank Adaptation (LoRA) mitigate computational burdens, their efficacy is heavily constrained by the scarcity of high-quality, expert-annotated labels in the RS domain.

Existing semi-supervised learning (SSL) frameworks attempt to leverage unlabeled data but face three specific bottlenecks in RS:

Sensitivity to Initial Confidence: Threshold-based selection is prone to excluding informative samples or including high-confidence errors when initial zero-shot accuracy is low due to domain gaps.
Class Imbalance and Bias: Reliance on absolute thresholds often leads to the over-representation of easily distinguishable categories while systematically excluding complex or ambiguous classes, creating a self-reinforcing bias toward majority classes.
Architectural Limitations: Traditional SSL methods often rely on uni-modal architectures with linear heads, failing to exploit the rich, high-dimensional semantic priors and cross-modal alignment capabilities inherent in VLMs.

Methodology: SE-CLIP Framework
The authors propose Self-Evolutionary CLIP (SE-CLIP), a semi-supervised framework designed for recursive label mining in scene classification. The approach utilizes a dual-phase pipeline:

Architectural Adaptation (LoRA): The framework employs the CLIP foundation model as a feature extractor. To bridge the domain gap without full fine-tuning, Low-Rank Adaptation (LoRA) is injected into the frozen Transformer layers. Trainable bypass matrices ( $A$ and $B$ ) update the weights ( $W = W_0 + BA$ ), focusing adaptation on relevant features while preventing overfitting to sparse initial seeds.
Semantic Anchoring: Instead of a traditional linear classification head, the model uses fixed textual anchors ( $T_c$ ) generated by encoding class-specific prompts (e.g., "an aerial photo of {c}") through the frozen text encoder. The visual representation is trained to align with these anchors using a cross-entropy loss over temperature-scaled cosine similarities.
Recursive Discovery Protocol:
- Warm-up: The model is first trained exclusively on a sparse set of expert-labeled seeds to stabilize gradients.
- Recursive Mining: Following the warm-up, the system enters an iterative discovery phase. For each class, samples in the unlabeled pool are ranked by their similarity to the class's textual anchor. A fixed budget ( $k$ ) of top-ranked samples is migrated to the labeled support set.
- Class-Balanced Selection: Crucially, the framework enforces a uniform budget $k$ across all categories. This prevents the optimization process from being dominated by high-frequency or easily discriminable classes, ensuring an equidistributed expansion of the support set.
- Self-Training: The discovered samples are appended to the labeled pool, and the LoRA parameters are re-optimized. This cycle repeats, allowing the model to recursively evolve its feature space.

Key Contributions

Recursive Discovery over Static Thresholding: Replaces rigid, heuristic thresholding with a recursive protocol anchored in CLIP's semantic latent space, utilizing fixed textual anchors to guide label discovery.
Class-Balanced Migration Strategy: Introduces a mechanism that enforces strict equidistribution of mined samples across the label space, mitigating the bias toward majority classes common in traditional SSL.
Integration of VLMs with SSL: Demonstrates how to effectively leverage the cross-modal alignment of VLMs for semi-supervised adaptation, moving beyond uni-modal architectures.

Experimental Results
The framework was evaluated on two benchmarks: UC Merced (UCM, 21 classes) and NWPU-RESISC45 (NWPU, 45 classes), using only 5 labeled seeds per class.

Performance: SE-CLIP achieved state-of-the-art results, reaching 99.09% accuracy on UCM and 95.07% on NWPU. This represents a significant improvement over the strongest existing baseline (DARP), which scored 95.87% on UCM and 88.86% on NWPU.
Ablation Studies:
- Recursive Mechanism: Removing the recursive discovery phase (relying only on warm-up) caused a drastic performance drop (e.g., from 95.07% to 76.34% on NWPU), confirming the necessity of continuous sample assimilation.
- Balanced Mining: Replacing the per-class balanced strategy with global mining led to catastrophic failure (42.33% on NWPU), validating that uniform category distribution is essential for stability.
- LoRA Rank: Performance improved monotonically with higher LoRA ranks, suggesting that increased parameter capacity aids in adapting to complex spatial textures.
Efficiency: The framework optimizes only 1.16% of the total CLIP parameters. Inference latency is low at 10.26 ms per batch (size 16), as the recursive mining modules are discarded during deployment.
Visualization: t-SNE visualizations confirmed that while the zero-shot CLIP baseline exhibits overlapping class distributions, SE-CLIP successfully reorganizes the embedding space into tightly bound, well-separated semantic clusters.

Significance and Claims
The paper posits that SE-CLIP provides a viable solution for adapting VLMs to the remote sensing domain with minimal human intervention. The authors claim that by combining parameter-efficient low-rank adaptation with a balanced, recursive pseudo-labeling strategy, the framework overcomes the limitations of traditional SSL methods in handling domain shifts and class imbalances. The results suggest that target-directed low-rank mining offers a more effective alternative for handling complex RS data distributions than standard semi-supervised frameworks.

The authors acknowledge limitations, noting that the framework requires expressive text vocabularies to construct discriminative anchors and may struggle in fine-grained scenarios where semantic descriptions cannot fully resolve high inter-class structural similarity. Future work is suggested to focus on developing robust VLM-driven paradigms for image classification using only class names.

Technical Summary: Self-Evolutionary CLIP (SE-CLIP) for Remote Sensing

More like this