🤖 AI

Reference-based Category Discovery: Unsupervised Object Detection with Category Awareness

This paper proposes Reference-based Category Discovery (RefCD), an unsupervised object detection framework that achieves category-aware detection without manual annotations by leveraging feature similarity between predicted objects and unlabeled reference images to guide the learning of category-specific features.

Original authors: Yichen Li, Qiankun Liu, Ying Fu

Published 2026-05-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yichen Li, Qiankun Liu, Ying Fu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to find things in a room.

The Old Way (Supervised Learning):
Traditionally, to teach a robot to spot a "dog," you have to show it thousands of photos of dogs and manually draw a box around every single one, writing "dog" on the label. This is like hiring a team of teachers to spend years labeling every picture in a library. It's expensive, slow, and if the robot hasn't seen a specific type of dog (like a Chihuahua), it might get confused.

The "Unsupervised" Way (The Current Problem):
Some researchers tried to teach robots without any labels at all. They said, "Just look at the pictures and find the shapes that look like objects." The robot gets good at finding something (like a blob of pixels), but it has no idea what that thing is. It's like a security guard who can tell you "there is a person in the hallway" but can't tell you if it's your mom, your neighbor, or a stranger. They are "category-agnostic" (blind to the specific type).

The "One-Shot" Way (The Other Problem):
Other methods try to teach the robot by showing it just one picture of a dog and saying, "Find all the dogs like this." But here's the catch: to make this work, the robot usually still needs a massive amount of pre-labeled data to learn its basic rules first. It's like trying to teach someone a new language by showing them one word, but they still need to have studied the dictionary for years beforehand.

The New Solution: RefCD (Reference-Based Category Discovery)

This paper introduces a new method called RefCD. Think of it as a "Look-Alike" game played without a rulebook.

The Core Idea:
Instead of using a dictionary of labels (like "dog," "cat," "car"), RefCD uses Reference Images.

The Analogy: Imagine you are at a party and you want to find everyone wearing a "red hat." You don't need to know the word "red hat" or have a list of names. You just hold up a photo of a red hat (the reference image) and say, "Find everyone who looks like this."
How it works: The robot looks at the target image and compares every object it sees to your reference photo. If the features (the visual "fingerprint") match closely, it says, "Aha! This is one of those!"

The Magic Ingredient: Feature Similarity Loss
The paper's biggest innovation is a new mathematical rule called Feature Similarity (FS) Loss.

The Metaphor: Imagine the robot is trying to learn a dance. In the past, a teacher would correct the robot by saying, "No, that's the wrong step for a 'Dog' dance."
RefCD's approach: There is no teacher with a list of steps. Instead, the robot is given a "Reference Dancer" (the reference image). The robot is told: "Your goal is to make your dance moves look exactly like the Reference Dancer's moves."
If the robot sees a dog in the picture, and the Reference Image is a dog, the robot learns to make its internal "dance moves" (features) match the reference dog. If it sees a cat, the moves won't match.
The Result: The robot learns to group things by how similar they look to the reference, without ever needing to know the word "dog" or "cat." It discovers categories on its own by matching patterns.

What Can It Do?

Find Specific Things (Category-Aware): You give it a photo of a specific type of shoe, and it finds all shoes like that in a messy room. It does this without any pre-trained labels.
Find Anything (Category-Agnostic): Even if you don't give it a reference photo, the training process makes it better at just finding any object in the image, acting like a standard object detector.
Track Moving Objects: The paper also shows it can follow a specific object (like a specific dog) as it moves across a video, similar to a "follow the leader" game.

The Results
The authors tested this on standard computer vision datasets (like COCO and ImageNet). They found that:

RefCD works better than previous "no-label" methods at finding specific types of objects.
It is surprisingly competitive with methods that do use expensive human labels.
It can even distinguish between very similar things (like a red apple vs. an orange apple) if you give it a clear reference photo, something many other robots struggle with.

In Summary
RefCD is a new way to teach computers to find things. Instead of memorizing a dictionary of labels, it learns by playing a game of "spot the similarity" using reference photos. It's a smarter, more flexible way to teach robots to see the world without needing a human to label every single picture first.

Technical Summary: Reference-Based Category Discovery (RefCD)

Problem Statement

Existing object detection paradigms face a dichotomy between supervision and generalization. Traditional supervised methods rely heavily on labor-intensive manual annotations and are confined to predefined categories, limiting their scalability to unseen objects. Conversely, unsupervised object detection methods (e.g., TokenCut, CutLER) successfully localize foreground objects without labels but lack category awareness; they treat all detected objects as a single foreground class, rendering them incapable of classification or distinguishing between different object types.

Meanwhile, one-shot and open-vocabulary detection methods attempt to address novel categories but typically require manual category labels for fine-tuning or rely on large, pre-trained vision-language models with significant computational costs. There remains a gap for a method that can perform category-aware detection (identifying specific object types) in a fully unsupervised manner without any manual annotations or predefined category lists.

Methodology: Reference-Based Category Discovery (RefCD)

The authors propose RefCD, an unsupervised detector that leverages unlabeled reference images to guide the learning of category-aware features. The framework operates without explicit category labels, instead using feature similarity to associate detected objects with reference instances.

1. Architecture Overview

RefCD consists of two primary components:

Reference Encoder: A frozen, self-supervised Vision Transformer (ViT, specifically DINOv2) used to extract feature representations from both the target image patches (pseudo boxes) and the user-provided reference images.
Detector: A transformer-based detector (RT-DETR) trained to perform two tasks simultaneously:
- Category-Agnostic Detection: Localizing all foreground objects using binary classification and regression losses.
- Category-Aware Matching: Identifying objects that share the same implicit category as the reference image via feature similarity.

2. Feature Similarity (FS) Loss

The core innovation of RefCD is the Feature Similarity (FS) loss, which replaces traditional classification losses (e.g., cross-entropy) that require ground-truth labels. The loss functions as follows:

Assignment: Instead of matching predictions to annotations based on category labels, RefCD uses the Hungarian algorithm with a cost function based on feature similarity (cosine similarity between query features and pseudo-box features) and bounding box distance.
Loss Calculation: The FS loss ( $L_{FS}$ $L_{F S}$ ) constrains the detector to maximize the similarity between a predicted query and a reference image if they belong to the same implicit category, and minimize it otherwise.
- For matched pairs where the pseudo-box corresponds to the reference image, the loss encourages the query's similarity to the reference to match the pseudo-box's similarity to the reference.
- For unmatched pairs, the loss penalizes high similarity.
Training Strategy: During training, pseudo boxes are generated from the target image using existing unsupervised methods (e.g., MaskCut). A random pseudo box is selected as the "reference" for that iteration. The model learns to associate other objects in the image with this reference if they share high feature similarity.

3. Inference

At inference, the model does not perform explicit classification. Instead, it accepts a user-provided reference image. The detector outputs category-agnostic boxes, and the system filters/identifies objects by matching their query features against the reference image's features. Objects with high feature similarity are considered matches. This allows for open-set detection where the "category" is defined dynamically by the reference image rather than a fixed label set.

Key Contributions

Unsupervised Reference-Based Detection: The proposal of RefCD, the first unsupervised detector capable of category-aware detection without any manual annotations, relying solely on reference images and feature similarity.
Feature Similarity Loss: The introduction of a novel loss function that guides the learning of potential category-specific features in an unsupervised paradigm, bridging the gap between category-agnostic localization and category-aware matching.
Unified Framework: RefCD supports both category-agnostic detection (localizing all objects) and category-aware detection (identifying specific instances) within a single framework, with the latter enhancing the former through better feature learning.

Experimental Results

The authors evaluated RefCD on standard benchmarks, demonstrating state-of-the-art performance in unsupervised settings.

Category-Aware (One-Shot) Detection:
- On COCO NOVEL (20 novel categories), RefCD achieved 15.1% AP, outperforming the previous state-of-the-art supervised method UNICL-SAM (13.2% AP) and significantly surpassing other supervised one-shot methods like Meta R-CNN (1.5% AP).
- On GMOT-40 (video tracking/detection), RefCD achieved 31.8% AP50 and 30.2% AR, demonstrating robustness in distinguishing objects of the same category across video frames.
Category-Agnostic Detection:
- On COCO 20K and val2017, RefCD achieved 13.3% AP and 12.9% AP, respectively. This outperforms leading unsupervised methods like CutLER (10.1% AP) and approaches the performance of CuVLER, despite using the same pseudo-box generation pipeline.
Extensions:
- Weakly Supervised Learning: When trained with only bounding box labels (no category labels) on COCO BASE, RefCD achieved 24.9% AP on COCO NOVEL, outperforming fully supervised one-shot methods.
- Single Object Tracking (SOT): Applied to the VOT 2018 dataset, RefCD demonstrated competitive performance in tracking instances without explicit labels.

Significance and Claims

The paper claims that RefCD successfully eliminates the reliance on manual annotations for category-aware detection, a limitation that has hindered both traditional unsupervised methods and one-shot approaches. By shifting the paradigm from "predicting a label" to "matching a reference via features," RefCD enables open-set detection where the system can identify novel objects simply by providing an example.

The authors emphasize that their method does not require pre-defined category lists or fine-tuning on labeled data. The results suggest that learning feature similarity in an unsupervised manner not only enables category discrimination but also implicitly improves the quality of category-agnostic localization (bounding box regression). The work positions itself as a unified framework that bridges the gap between unsupervised object discovery and reference-based detection, offering a scalable solution for detecting objects in scenarios where annotations are unavailable or impractical.