World Model Failure Classification and Anomaly Detection for Autonomous Inspection

1. Problem Statement

Autonomous inspection robots are increasingly used in industrial and office environments to monitor instruments (e.g., pressure gauges) to reduce human risk and cost. However, these robots face significant challenges in ensuring accurate readings due to:

• Occlusions: Instruments blocked by other equipment.
• Limited Viewpoints: Poor angles or truncated fields of view.
• Environmental Conditions: Glare, shadows, and lighting changes that fall outside the robot's training distribution.

Current systems often struggle to distinguish between known failures (e.g., a systematic misread due to occlusion) and Out-of-Distribution (OOD) anomalies (e.g., unexpected environmental noise). Existing methods either focus solely on anomaly detection (providing no specific corrective action) or supervised classification (failing to detect novel, unseen failures). The core challenge is to develop a system that can detect failures in real-time, classify them as known or unknown, and do so faster than a human observer to enable immediate intervention.

2. Methodology

The authors propose a hybrid framework that combines supervised failure classification with sequential OOD detection, utilizing a World Model backbone and Conformal Prediction (CP) for thresholding.

A. Framework Architecture

The system operates as a decision function $D(\tau_t)$ that classifies a robot's trajectory $\tau_t$ into one of three categories:

1. Success: Accurate reading obtained.
2. Known Failure: A recognized failure mode (e.g., occlusion) requiring specific corrective action.
3. Anomaly (OOD): Unseen conditions (e.g., extreme glare) where the robot cannot produce a reading but it is not necessarily a "failure" in the traditional sense, just an inability to proceed.

The classification logic relies on two parallel decision functions:

• $D_{success}$: Trained on successful data.
• $D_{fail}$: Trained on known failure data.

A trajectory is classified as Anomaly (OOD) only if both models indicate a deviation (i.e., both return a score above their respective thresholds). Otherwise, it is classified based on which specific threshold is breached.

B. World Model & Scoring

• Backbone: The system uses the NVIDIA Cosmos Tokenizer and a latent world model to predict future video frames based on current inputs.
• Input: Compressed video feeds (512x288) from a Pan-Tilt-Zoom (PTZ) camera.
• Scoring: Instead of relying on action-dependent metrics, the system calculates prediction errors (residuals) between the predicted next frame and the actual observed frame.
• Metrics Evaluated: The paper tests various scoring methods, including Reconstruction Error, Latent Prediction Error, Latent Standard Deviation, Mahalanobis Distance, and Latent Cosine Distance.

C. Conformal Prediction (CP) Thresholding

To ensure the system is policy-agnostic and distribution-free, the authors use Conformal Prediction to calibrate decision thresholds ($\eta_s$ and $\eta_f$).

• Calibration: Separate calibration sets for "Success" and "Failure" are used to determine quantile-based thresholds.
• Mechanism: If a trajectory's score exceeds the calibrated threshold, it is flagged. This allows the system to guarantee a specific coverage rate for in-distribution data while flagging deviations.
• Static Bands: The authors use constant-value bands rather than time-varying ones to simplify real-time deployment.

3. Key Contributions

1. Hybrid Detector: A unified framework that simultaneously performs supervised classification of known failures and unsupervised detection of OOD anomalies using a single world model backbone.
2. Calibrated Thresholds: The application of Conformal Prediction to derive robust, statistically grounded thresholds for both success and failure models, enabling reliable separation of three distinct classes.
3. Real-World Validation: Evaluation on a dataset of 290 videos collected from both office and industrial sites, covering diverse gauge types and environmental conditions.
4. Hardware Deployment: Successful real-time deployment on a Boston Dynamics Spot robot, demonstrating the system's ability to trigger interventions (e.g., re-zooming, re-attempting) based on classification.

4. Experimental Results

The framework was evaluated on accuracy and detection time (latency compared to human observation).

• Accuracy:

  • The Latent Prediction Error metric achieved the best balance, reaching >90% accuracy across all three classes (Success, Known Failure, OOD) at the 90% quantile.
  • Mahalanobis Distance showed perfect separation (100% accuracy) but suffered from overfitting and instability in high-dimensional spaces, leading to false positives.
  • Reconstruction Error and Cosine Distance performed poorly, likely because the pre-trained Cosmos model was already optimized for reconstruction, leaving little residual signal for anomalies.

• Detection Time:

  • The system consistently detected failures earlier than human observers.
  • Using Latent Prediction Error, the system classified failures on average 1.8 to 3.5 seconds earlier than a human could visually confirm the issue.
  • This anticipatory capability is attributed to the model detecting subtle latent pattern deviations before they become visually obvious.

• Hardware Performance:

  • Deployed on a Boston Dynamics Spot with an external inference server (due to bandwidth/storage constraints during testing).
  • The pipeline requires ~1.5 GB of storage (mostly pre-trained weights) and can theoretically run onboard.
  • Classification latency was dominated by network transfer, but the scoring itself was instantaneous.

5. Significance and Future Work

• Safety & Reliability: This work addresses a critical gap in autonomous systems: the ability to not only detect that something is wrong but to categorize the error. This allows for specific, informed interventions (e.g., "move closer" for occlusion vs. "wait for better light" for glare) rather than generic "stop" commands.
• Data Quality Feedback: The system can identify OOD cases, which serves as a feedback loop to guide targeted data collection for retraining, improving the robot's robustness over time.
• Future Directions: The authors suggest moving toward adaptive, time-varying CP bands to handle distribution shifts (e.g., seasonal changes), incorporating semantic understanding for higher-level failure recognition, and training a single unified model with a classification head to reduce computational overhead.

In summary, this paper presents a robust, real-time solution for autonomous inspection that leverages world models and statistical calibration to anticipate and classify failures, significantly outperforming human reaction times in safety-critical industrial scenarios.