Identifying the Unknown: Prompt-Free Open Vocabulary Anomaly Recognition for Robot-Object Interaction

Technical Summary: AnomNOVIC – Prompt-Free Open Vocabulary Anomaly Recognition

Problem Statement
Robotic agents operating in real-world, unstructured environments face a critical bottleneck: the inability to visually perceive and recognize previously unseen objects in real-time. While Large Language Models (LLMs) have advanced high-level reasoning, they rely on visual perception systems that are often constrained. Existing open-vocabulary object detectors generally fall into two categories with significant limitations:

1. Prompted Models: Systems like ViLD, OWLv2, and YOLO-World require a predefined list of candidate object classes at inference time. This "prompting" constraint severely limits practical utility, as a robot cannot detect objects it has not been explicitly instructed to look for, and these models often fail ungracefully (producing false positives) when encountering novel objects.
2. Limited Openness: Even "prompt-free" models often suffer from limited active vocabularies due to training on datasets with restricted concept coverage (e.g., YOLOE relies on a fixed vocabulary of ~4,585 tags). Furthermore, many existing models cannot simultaneously localize multiple objects and assign fine-grained, free-form labels, a capability crucial for robotic manipulation.

Methodology: The AnomNOVIC Framework
The authors propose AnomNOVIC, a two-stage, prompt-free framework designed for known workspaces (e.g., a robot's tabletop). The system decouples object localization from classification to achieve unbiased detection of any novel object.

• Stage 1: Class-Agnostic Localization via Anomaly Detection

  • Architecture: A Vision Transformer (ViT) Masked Autoencoder (MAE) is trained exclusively on images of the robot's empty workspace (the "normative baseline").
  • Training Strategy: The MAE is trained to reconstruct the scene. To teach the model to identify objects as anomalies, an "object pasting" augmentation strategy is employed: synthetic anomalies are pasted into the training images but omitted from the target reconstruction. Consequently, the MAE learns to reconstruct the background while ignoring (and thus flagging) the pasted objects.
  • Outputs: The decoder is extended to produce three aligned outputs: a reconstructed RGB image, a binary table mask, and an anomaly mask.
  • Inference: During inference, objects present on the table are treated as anomalies because they were not in the training distribution. The system computes a reconstruction error map and an anomaly mask. These are merged, filtered, and processed (optionally refined by SAM 2.1) to generate class-agnostic bounding boxes for all detected objects.

• Stage 2: Prompt-Free Classification via NOVIC

  • Architecture: The cropped regions of interest (RoIs) identified in Stage 1 are passed to NOVIC, a real-time, open-vocabulary image classifier.
  • Mechanism: Unlike prompted models, NOVIC generates free-form object noun labels without requiring a candidate class list. It leverages a pretrained CLIP model and a classification decoder trained on a massive, content-controllable synthetic text dataset covering a near-unconstrained taxonomy of object nouns (~43K).
  • Scoring: The final confidence score combines the NOVIC prediction probability with the anomaly confidence derived from the MAE's reconstruction error and mask outputs.

Key Contributions

1. Prompt-Free Open Vocabulary Detection: AnomNOVIC is the first framework to successfully integrate a powerful prompt-free classifier (NOVIC) with a class-agnostic localization pipeline, enabling the detection and labeling of objects without any predefined class lists.
2. Anomaly-Based Localization: By framing object detection as an anomaly detection task within a known workspace, the system avoids vocabulary bias. It utilizes the principle that "anything that does not belong" is an object, allowing for the discovery of objects of any shape, texture, or novelty.
3. Dataset and Benchmarking: The authors introduce a challenging dataset collection for the NICOL humanoid robot, including "plain" and "distractor" tabletop scenarios, as well as a "Wild" test set with 48 unique, in-the-wild objects.
4. Model Adaptation: The work details the retraining and frequency-thresholding of NOVIC variants (N-FT0, N-FT2) specifically tuned for the robot's laboratory environment, demonstrating that dataset reweighting improves performance in specific open-set scenarios.

Experimental Results
Evaluated on the NICOL tabletop environment and the Wild dataset, AnomNOVIC significantly outperforms state-of-the-art baselines, including YOLOE (the only available prompt-free baseline), OWLv2, and YOLO-World-v2.

• Prompt-Free Performance: On the plain dataset, AnomNOVIC-S (with SAM refinement) achieves 47.1% AP and 57.5% AP50, compared to YOLOE's 20.4% AP. On the Wild dataset, it achieves up to 82.6% detection and classification accuracy, vastly surpassing YOLOE's 14.2%.
• Prompted Performance: When class candidates are provided, AnomNOVIC reaches 59.0% AP and 72.5% AP50, again doubling the performance of the strongest prompted competitors (e.g., YOLO-World-v2-X).
• Qualitative Superiority: The system demonstrates the ability to provide fine-grained, correct classifications (e.g., "silver medal," "audio amplifier") where prompted baselines frequently produce high-confidence false positives or fail to detect novel items.

Significance and Claims
The paper claims that AnomNOVIC addresses the "strong constraint" of prompted models, which undermines the core advantage of open-vocabulary detection by limiting robots to only known classes. By decoupling localization from classification and utilizing anomaly detection, the system enables robots to interact with "totally novel objects at runtime."

The authors position this work as a step toward "semantically aware robots" capable of perceiving and interacting with open-ended environments. They emphasize that the approach is particularly effective for known workspaces (like a robot's table) where a normative baseline can be established, allowing for unbiased, real-time discovery of the unknown without the need for task-specific fine-tuning or class prompts. The results suggest that combining reconstruction-based anomaly detection with unconstrained classification is a viable path toward true open-world autonomy in robotics.