💻 computer science

SECOS: Semantic Capture for Rigorous Classification in Open-World Semi-Supervised Learning

The paper introduces SECOS, a novel approach for open-world semi-supervised learning that leverages external knowledge to extract and align semantic representations, enabling models to directly predict semantically relevant textual labels for both known and novel classes without post-processing, thereby outperforming existing methods.

Original authors: Hezhao Liu, Jiacheng Yang, Junlong Gao, Mengke Li, Yiqun Zhang, Shreyank N Gowda, Yang Lu

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hezhao Liu, Jiacheng Yang, Junlong Gao, Mengke Li, Yiqun Zhang, Shreyank N Gowda, Yang Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are running a school for a new type of animal classification. You have a textbook with pictures and names of animals you already know (like Dogs and Cats). However, you also have a huge pile of mystery photos from a safari. Some of these photos are of the animals you know, but many are of new, unknown animals (like a Jellyfish or a Kangaroo) that aren't in your textbook.

Your goal is to teach a student (the AI model) to look at any photo and immediately say, "That's a Dog," "That's a Kangaroo," or "That's a Jellyfish," using the correct names from a master list.

The Problem: The "Guessing Game" of Old Methods

Previous methods tried to solve this, but they played a rigged game.

The Training: They focused so much on the visual details of the animals they knew (the textbook) that they ignored the mystery animals.
The Test: When the student got a picture of a Kangaroo, they might guess "Animal #3." Since the teacher didn't know what "Animal #3" meant, they used a trick called Hungarian Matching. This is like a game of "Musical Chairs" where, after the test, you rearrange the answers to make the score look perfect.
- Example: The student guessed "Animal #3" for a Kangaroo. The teacher says, "Okay, let's pretend 'Animal #3' actually means 'Kangaroo'."
- The Flaw: In the real world, you can't rearrange the answers after the fact. You need the student to know the actual name ("Kangaroo") right away. Old methods were essentially clustering animals into groups without knowing what the groups were called.

The Solution: SECOS (The "Semantic Translator")

The authors created a new system called SECOS (Semantic Capture for Open-World Semi-Supervised Learning). Instead of just guessing numbers, SECOS acts like a translator that connects pictures directly to words.

Here is how SECOS works, using three simple steps:

1. The "Confident Guess" (Novel Class Semantic Compensation)

Since the student has no textbook for the new animals, SECOS uses a super-smart, pre-trained teacher (a massive AI that already knows the world) to take a quick look at the mystery photos.

The teacher says, "I'm 90% sure this looks like a Kangaroo."
SECOS takes only the photos where the teacher is very confident and creates a mini-textbook for the new animals. This balances the learning so the student doesn't just ignore the new animals.

2. The "Double-Check" (Batch-Wise Semantic Recapture)

Sometimes, a photo is tricky. Is it a Dog or a Wolf?

SECOS looks at a whole batch of photos at once. It checks two things:
1. Does this photo look like a specific animal? (Instance-to-Class)
2. Do other photos in this batch also look like this animal? (Class-to-Instance)
If a photo passes both checks (e.g., it looks like a Kangaroo, and other photos in the group also look like Kangaroos), SECOS gives it a "gold star" and a solid label. This filters out the confusing, blurry guesses and keeps only the clear, high-quality examples.

3. The "Bridge" (Adapter for Semantic Feature Alignment)

Now the student has a pile of pictures and a pile of names, but they speak different languages (pixels vs. words).

SECOS builds a special bridge (called an Adapter) between the picture part of the brain and the word part.
This bridge learns to say, "When I see these specific pixel patterns, it means the word 'Kangaroo'."
Crucially, this bridge allows the model to directly output the word "Kangaroo" instead of a random number like "Class 5."

Why This Matters

The paper claims that SECOS is the first to truly solve the "Rigorous Classification" problem.

No Rearranging: It doesn't need to shuffle answers after the test to get a high score. It gets the right name on the first try.
Better Results: Even when compared to old methods that were allowed to use the "Musical Chairs" trick (Hungarian matching) to boost their scores, SECOS still beat them by a significant margin (up to 5.4% better).
Real-World Ready: Because it predicts actual names, it works in real scenarios where you can't go back and fix the labels later.

The Bottom Line

Think of old methods as a student who can sort a pile of mixed-up toys into neat piles but can't tell you what the toys are called. SECOS is a student who can sort the toys and immediately shout out the correct names ("Ball!", "Teddy Bear!", "Robot!") without needing a cheat sheet or a second guess. It achieves this by using a smart teacher to fill in the blanks and a special bridge to connect pictures to words.

1. Problem Definition: The Gap in Open-World Semi-Supervised Learning (OWSSL)

Context:
Open-World Semi-Supervised Learning (OWSSL), also known as Generalized Class Discovery (GCD), involves training models on labeled data from "known" classes and unlabeled data containing both "known" and "novel" (previously unseen) classes.

The Core Limitation:
Existing OWSSL methods fail to perform Rigorous Classification (RC-OWSSL).

Lack of Semantic Grounding: Current methods focus heavily on visual feature clustering but ignore the latent semantic information linking visual features to linguistic meanings.
Post-Hoc Matching Dependency: During evaluation, existing methods rely on Hungarian matching to align predicted cluster IDs with ground-truth class names. This is a post-processing step that artificially inflates accuracy by reassigning labels to maximize overlap, regardless of whether the model actually learned the correct semantic correspondence.
Practical Impossibility: In real-world deployment, a labeled test set for Hungarian matching is unavailable. Therefore, existing methods cannot directly predict the correct textual label (e.g., "Dog" or "Cat") for a novel sample; they only output a cluster index.

Goal:
The paper defines RC-OWSSL, a task where the model must directly predict the most semantically relevant textual label from a candidate set (known + novel) without any post-hoc label reassignment.

2. Methodology: The SECOS Framework

The authors propose SECOS (SEmantic Capture for Open-world Semi-supervised learning), a framework designed to bridge the gap between visual features and textual semantics using external knowledge (CLIP). SECOS consists of three core modules:

A. Novel Class Semantic Compensation (Global Level)

Problem: Severe supervision imbalance; known classes have labels, novel classes do not, leading to a strong bias toward known classes.
Solution: Construct a Global Pseudo-Labeled Novel Dataset ( $D_N$ ).
1. A frozen "Teacher" CLIP model assigns hard pseudo-labels to unlabeled data.
2. Samples are grouped by these pseudo-labels.
3. The top- $\phi\%$ highest-confidence samples for each novel class are selected to form $D_N$ .
4. Semantic Enhancement: A Large Language Model (LLM) generates descriptive prompts for class names to create richer semantic representations, which are used to refine the pseudo-labeling process.
Outcome: This balances the supervision strength between known and novel classes globally.

B. Batch-Wise Semantic Recapture (Local/Instance Level)

Problem: Global compensation is coarse-grained; it misses fine-grained semantic nuances within specific batches and known classes.
Solution: A dual-perspective confidence filtering mechanism applied to batches of unlabeled data ( $B$ $B$ ).
1. Intra-instance: For a specific sample, accumulate confidence scores across classes until a threshold ( $\tau$ ) is met.
2. Inter-instance: For a specific class, determine a threshold ( $\theta$ ) based on the confidence distribution of all samples in the batch.
3. Selection: A sample is selected as a high-quality pseudo-labeled instance only if it satisfies both criteria and the intersection of candidate labels results in exactly one label.
Outcome: This filters out ambiguous samples and captures fine-grained semantic information, providing high-precision supervisory signals.

C. Adapter for Semantic Feature Alignment

Problem: Even with pseudo-labels, the model needs to align its visual feature space with the semantic space of the textual labels to perform direct classification.
Solution: A lightweight Adapter mechanism inserted into a frozen CLIP visual encoder.
1. Architecture: Parallel adapters are inserted after attention blocks in the visual encoder. The text encoder remains frozen.
2. Alignment: The model computes similarity between the visual features (processed by the adapter) and the semantic features of the candidate textual labels (generated from class descriptions).
3. Training: Only the adapter and projector parameters are updated. The loss is a standard Cross-Entropy loss between the predicted logit and the pseudo-label (or ground truth).
Outcome: The model learns to map visual inputs directly to textual labels, enabling rigorous classification without Hungarian matching.

3. Key Contributions

Definition of RC-OWSSL: The paper formally identifies the intrinsic deviation between existing OWSSL (clustering) and the practical requirement of Rigorous Classification (direct textual prediction), highlighting the failure of current methods to achieve semantic grounding.
SECOS Framework: Proposes a novel framework that leverages external knowledge (CLIP + LLM) to extract and align semantic representations. It uniquely addresses the lack of supervision for novel classes through global compensation and batch-wise recapture.
Superior Performance: Demonstrates that SECOS outperforms State-of-the-Art (SOTA) methods even when those methods are evaluated under the lenient "post-hoc matching" setting, while SECOS operates under the stricter "direct classification" setting.

4. Experimental Results

Datasets:
Evaluated on 7 datasets: 3 generic (CIFAR-10, CIFAR-100, ImageNet-100) and 4 fine-grained (CUB, Stanford Cars, Oxford Flowers, Oxford Pets).

Performance Highlights:

Direct Comparison: SECOS achieves significant improvements over SOTA methods (e.g., TextGCD, TP-OWSSL, SimGCD) even though SECOS does not use Hungarian matching.
Quantitative Gains:
- On generic datasets, SECOS improves Novel accuracy by up to 7.37% and All accuracy by 5.35% compared to the best existing methods.
- On fine-grained datasets, it shows improvements of 1.78% (Novel) and 1.95% (All) over TP-OWSSL.
- Overall, SECOS surpasses existing methods by up to 5.4% in rigorous classification metrics.
Ablation Studies:
- Removing either the Global Compensation or Batch-Wise Recapture modules causes a drastic drop in Novel class accuracy (e.g., dropping from 82.6% to 14.4% on CIFAR-100 without any semantic capture).
- The method is robust to batch size variations and adapter downsampling dimensions.
Self-Learning: SECOS can function effectively even without a separate "Teacher" model (using an EMA-updated student), showing only a minor performance drop (~0.83%) while still outperforming teacher-dependent baselines.

5. Significance and Impact

Paradigm Shift: The paper challenges the community to move away from "clustering-based" evaluation (Hungarian matching) toward "classification-based" evaluation, which is the only viable approach for real-world deployment where ground-truth labels for novel classes are unknown.
Semantic Grounding: It establishes that successful OWSSL requires explicit alignment between visual structures and linguistic semantics, not just visual feature clustering.
Practical Applicability: By enabling direct textual prediction, SECOS makes open-world learning applicable to real-world scenarios (e.g., autonomous driving, medical diagnosis) where models must identify and name new categories on the fly without re-labeling the test set.
Efficiency: The use of lightweight adapters allows the model to leverage powerful pre-trained CLIP knowledge without the computational cost of full fine-tuning.

In summary, SECOS solves the critical bottleneck in open-world learning by introducing a mechanism to "capture" latent semantics, allowing models to rigorously classify novel objects into their correct textual categories directly.