💻 computer science

Towards Sparsely Annotated Open-World Object Detection

This paper introduces Sparsely Annotated Open-World Object Detection (SA-OWOD), a new task addressing the co-occurrence of sparse supervision and unseen categories, and proposes the Dual-Perspective Object Discovery (DPOD) framework to effectively resolve ambiguous supervision signals and improve the detection of both known and unknown objects.

Original authors: HeeJu Han, AJeong Kim, Jinsun Park

Published 2026-08-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: HeeJu Han, AJeong Kim, Jinsun Park

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to recognize animals in a forest. In a perfect classroom, you would show the robot a picture of a dog, a cat, and a bird, and you would circle exactly where they are. The robot learns easily because everything is labeled. But in the real world, things are messy. Sometimes, you might forget to circle a dog in a photo because you were in a hurry (this is called sparse annotation). Other times, the robot might see a creature it has never seen before, like a platypus, and you don't even know what to call it (this is the open-world problem).

For a long time, scientists tried to solve these two problems separately. One group taught robots to guess where the missing dogs were, assuming everything in the picture was a known animal. Another group taught robots to spot new, unknown creatures, but they assumed the robot had perfect labels for everything it did know. The problem is, real life is a mix of both: you might have a photo with a missing label for a known dog and a hidden platypus. If the robot isn't taught to handle this confusion, it might think the unlabelled dog is a new mystery creature, or it might ignore the platypus entirely. This paper tackles that specific, confusing middle ground.

The researchers, HeeJu Han, AJeong Kim, and Jinsun Park, introduce a new challenge they call Sparsely Annotated Open-World Object Detection (SA-OWOD). Think of this as a detective game where the detective has a blurry photo with some clues missing and some strange, unidentifiable figures in the background. The goal is to figure out which blurry spots are actually known animals that were just missed by the photographer, and which ones are truly new, unknown creatures.

To solve this, they built a new system called DPOD (Dual-Perspective Object Discovery). Imagine DPOD as a detective with two special tools working together. The first tool is the Known Target Recovery Module (KTRM). This tool acts like a "second look" specialist. It scans the photo again, looking for things that look like the animals the robot already knows (like dogs), even if the original photo didn't have a label for them. It says, "Hey, that shape looks exactly like a dog we've seen before; let's give it a label so we don't ignore it." This helps the robot stop treating missing dogs as background noise.

The second tool is the Dual-Disagreement Target Generator (DDTG). This tool is a bit more skeptical. It looks at the photo from two slightly different angles (or "views"), kind of like how our two eyes see the world. If the robot is confident about a dog, it will agree on the dog's identity from both angles. But if the robot sees a strange, unknown creature, it might get confused and say, "Is that a bird? Or a squirrel?" from one angle, and "No, maybe a cat?" from the other. DDTG spots these moments of confusion. It says, "If the robot is arguing with itself about what this thing is, it's probably something new and unknown!" This helps the robot find the mystery creatures without getting tricked by the missing labels.

When the researchers tested their system, they found that it worked much better than previous methods, especially when the labels were very sparse. In their experiments, they created a "hard" scenario where they removed about half of the labels from the training images. Even in this difficult setting, their method was able to find 51.85% of the unknown objects (a metric called U-Recall), which was a significant jump compared to other top methods that struggled to find more than 33% in the same conditions. They also showed that their system didn't get confused; it kept its ability to find the known animals (like dogs and cats) just as well as before, maintaining a high score of 54.09 on known detection.

The paper suggests that by using these two tools together—one to recover the missing known items and one to spot the confused unknown items—the robot can finally handle the messy, real-world situation where labels are missing and new things appear all at once. While the system is a big step forward, the authors note that it still relies on some fixed rules for deciding what counts as "confusing" or "missing," which might need to be adjusted for different types of objects in the future. But for now, DPOD offers a promising way to teach robots to see the world more like humans do: with a mix of known facts and open-minded curiosity.

Technical Summary: Towards Sparsely Annotated Open-World Object Detection

1. Problem Definition: Sparsely Annotated Open-World Object Detection (SA-OWOD)

Real-world object detection faces two simultaneous challenges: incomplete supervision (where many object instances are unlabeled due to annotation costs) and open-world conditions (where unseen categories appear). While Sparsely Annotated Object Detection (SAOD) and Open-World Object Detection (OWOD) have been studied independently, they fail when these conditions co-occur.

SAOD Limitations: Existing SAOD methods assume a closed-world setting. They treat unlabeled regions as missing annotations of known classes, inadvertently suppressing or misclassifying truly unknown objects as background.
OWOD Limitations: Existing OWOD methods assume dense, fully annotated training data. Under sparse annotations, unlabeled known objects are treated as unknown or background, blurring the decision boundary between known and unknown classes and degrading detection accuracy for both.

The authors introduce SA-OWOD, a new task formulation where unlabeled regions in an image may correspond to either unlabeled known objects (missing annotations) or genuinely unknown objects. The core challenge is distinguishing between these two indistinguishable unlabeled regions without explicit supervision.

2. Methodology: Dual-Perspective Object Discovery (DPOD)

To address the ambiguity of unlabeled regions, the paper proposes DPOD, a unified framework comprising two complementary modules: the Known Target Recovery Module (KTRM) and the Dual-Disagreement Target Generator (DDTG).

A. Known Target Recovery Module (KTRM)
KTRM aims to recover supervision for unlabeled known objects and enforce feature separation between known and unknown categories.

Pseudo-Label Recovery: Inspired by Co-Student, KTRM uses a teacher-student framework with multiple augmented views to generate reliable pseudo-labels for unlabeled known objects. These pseudo-labels are combined with sparse ground truth to compensate for missing annotations.
Feature Separation: To prevent unlabeled known objects from overlapping with unknown instances in the embedding space, KTRM employs a Facility-Location conditional gain objective. This loss encourages known proposals (both labeled and pseudo-labeled) to move away from their nearest unknown counterparts in the feature space, stabilizing decision boundaries.

B. Dual-Disagreement Target Generator (DDTG)
DDTG identifies reliable unknown candidates by exploiting semantic instability across different views.

Cross-View Disagreement: The module generates two views of the same image and projects object proposals from a reference view to the target view to ensure geometric alignment. It then computes the cosine similarity between the classification logit vectors of the two views.
Candidate Selection: Proposals with high objectness scores but low semantic similarity (high disagreement) between views are selected as unknown candidates. This leverages the observation that unknown objects often reside near ambiguous decision boundaries, leading to unstable predictions under view perturbations.
Loss Function: DDTG applies classification loss to these disagreement candidates and a separation loss to push them away from known features, reinforcing the discovery of novel classes.

Overall Objective: The total loss combines the KTRM and DDTG losses, balancing the recovery of known objects and the discovery of unknown ones.

3. Key Contributions

Task Formulation: The authors are the first to formulate the SA-OWOD task, explicitly addressing the co-occurrence of sparse annotations and open-world scenarios.
Novel Framework (DPOD): They propose a dual-perspective framework that resolves ambiguity by:
- Recovering missing known annotations via pseudo-labeling (KTRM).
- Identifying unknown objects via cross-view semantic inconsistency (DDTG).
- Enforcing feature-level separation to stabilize decision boundaries.
Benchmarks and Evaluation: The paper constructs sparsely annotated open-world benchmarks based on Pascal VOC and MS-COCO, defining specific protocols (Easy, Hard, Coco50missp, Keep1, Extreme) to evaluate both known object recovery and unknown object discovery.

4. Experimental Results

Experiments were conducted on the proposed benchmarks using a Faster R-CNN backbone. The method was compared against state-of-the-art OWOD methods (RandBox, PROB, OrthogonalDet, CROWD) retrained under identical sparse annotation protocols.

Performance: DPOD consistently outperforms existing methods across all sparsity levels (from Easy to Extreme).
- In the Hard setting, DPOD improved Unknown Recall (U-Recall) by 18.50 percentage points over CROWD (51.85 vs 33.35) and 29.64 percentage points over OrthogonalDet (51.85 vs 22.21) in Task 2.
- In the Extreme setting, the gap in U-Recall over OrthogonalDet was 23.05 percentage points (45.56 vs 22.51) in Task 2.
Robustness: While other methods suffered significant degradation in U-Recall as annotation sparsity increased, DPOD maintained high performance. For instance, in the Extreme setting, DPOD's U-Recall degradation relative to the fully annotated setting was 17.31%, whereas OrthogonalDet dropped by 43.84%.
Ablation Studies:
- Both KTRM and DDTG contribute positively; KTRM primarily stabilizes known detection (K-mAP), while DDTG enhances unknown discovery (U-Recall).
- The combination of both modules yields the best results, confirming their complementary nature.
- Sensitivity analysis showed that moderate objectness thresholds and sufficiently diverse semantic disagreement signals are critical for performance.

5. Significance and Limitations

Significance:
The paper claims that SA-OWOD provides a practical benchmark for detection under ambiguous supervision, bridging the gap between theoretical open-world detection and real-world annotation constraints. By explicitly modeling the interplay between missing known annotations and unknown categories, DPOD prevents the misclassification of known objects as unknown and vice versa, leading to more robust open-world detection.

Limitations:
The authors acknowledge that DPOD currently relies on fixed thresholds for objectness ( $\tau_o$ ) and similarity ( $\tau_s$ ). In open-world scenarios, object characteristics vary significantly across tasks (e.g., accessories like ties often have lower objectness scores). The authors note that enforcing fixed thresholds under such noisy conditions is suboptimal and may create bottlenecks in progressive learning, suggesting that task-adaptive thresholds are a promising direction for future work.