Multimodal Anomaly Detection for Human-Robot Interaction

1. Problem Statement

The integration of robots into human-centric environments (logistics, healthcare, manufacturing) necessitates robust Human-Robot Interaction (HRI) systems. While standard safety protocols address known risks, they struggle with anomalous behaviors—unforeseen events that deviate from normal operation and can lead to system failures or unsafe outcomes.

Current anomaly detection approaches in robotics face two main limitations:

• Data Scarcity: Anomalies are rare, making supervised learning difficult; thus, unsupervised methods are preferred.
• Input Modality: Most existing reconstruction-based models (e.g., autoencoders) operate on raw pixel data (video frames). This is computationally expensive, struggles to generalize in dynamic environments, and often reconstructs irrelevant details (background, lighting) rather than semantic actions.

The core problem addressed is how to detect anomalies in HRI efficiently and robustly by moving away from raw pixel reconstruction toward semantically meaningful feature reconstruction enhanced by multimodal data.

2. Methodology: The MADRI Framework

The authors propose MADRI (Multimodal Anomaly Detection for Human-Robot Interaction), a framework that performs anomaly detection in a feature space by fusing three distinct data modalities:

A. Visual Feature Extraction (Semantic Space)

Instead of reconstructing raw video frames, the system extracts high-level semantic features:

• Model: A pre-trained Swin3D model (originally trained on the Kinetics dataset) is fine-tuned on HRI action classes (e.g., Human Handover, Robot Picking, Place).
• Transformation: The final classification layer is removed. The model outputs 768-dimensional feature vectors representing the semantic content of video clips (32 frames).
• Benefit: This reduces computational load and focuses the reconstruction model on relevant motion patterns and object manipulations rather than background noise.

B. Sensor Data Integration

To capture internal robot states (e.g., mechanical stress, joint limits):

• Data: Joint torque readings and gripper positions are streamed from the robot.
• Processing: Max Pooling is applied across the temporal axis to downsample the high-frequency sensor data while preserving salient peaks.
• Fusion: These sensor features are concatenated with the visual feature vectors.

C. Scene Graph Representation

To capture relational context (relationships between humans and objects):

• Model: A pre-trained OED model generates a temporal Scene Graph where nodes are entities (humans, objects) and edges are relationships.
• Processing: The graph is converted into a matrix representing detected objects and their relations, which is then concatenated to the feature vector.

D. Reconstruction Model (Autoencoder)

• Architecture: A standard Autoencoder (Encoder-Decoder) with Linear layers, Batch Normalization, ReLU, and Dropout.
• Training: The model is trained only on normal data to learn the distribution of standard behavior.
• Inference: During operation, the model attempts to reconstruct the input feature vector.
  • Anomaly Scoring: The Reconstruction Error (L1 Loss) is calculated.
  • Thresholding: Errors exceeding a dynamically determined threshold (optimized via F1-Score) are flagged as anomalies.

3. Key Contributions

1. MADRI Framework: A novel multimodal approach combining video, robot internal sensors, and Scene Graphs for HRI anomaly detection.
2. Feature-Space Reconstruction: Demonstrates that performing reconstruction on semantic feature vectors (derived from Swin3D) is more effective and efficient than raw pixel reconstruction for video-based anomaly detection.
3. Custom Dataset: Creation of a specific Human-Robot Handover Dataset containing 72 recordings (6 participants, 12 trials each) with 17 naturally occurring failure cases (e.g., dropped cups, collisions, joint limit violations).
4. Multimodal Validation: Experimental proof that fusing modalities significantly outperforms vision-only baselines, particularly for detecting internal mechanical failures invisible to cameras.

4. Experimental Results

The authors conducted an ablation study comparing four configurations:

1. Vision-Only
2. Vision + Sensors
3. Vision + Scene Graph
4. Full Multimodal (Vision + Sensors + Scene Graph)

Key Findings:

• Vision-Only Effectiveness: Reconstruction on feature vectors alone is effective for detecting visual anomalies (e.g., a human dropping a cup).
• Sensor Modality Impact: Adding sensor data (joint torques) provided the most significant performance boost. It allowed the model to detect internal failures (e.g., the robot reaching joint limits) that were invisible to the camera.
• Robustness: The multimodal model reduced false positives. In cases where visual fluctuations caused the vision-only model to spike in error, the sensor data helped stabilize the reconstruction error, correctly identifying the event as normal.
• Scene Graph Limitation: Surprisingly, adding the Scene Graph degraded performance. The authors attribute this to the low accuracy of the Scene Graph generator and the fact that the specific anomalies in their dataset were not primarily "relational" failures.
• ROC Analysis: The Vision + Sensor model achieved the highest Area Under the Curve (AUC), clearly outperforming the vision-only baseline.

5. Significance and Conclusion

The paper establishes that multimodal feature reconstruction is a superior strategy for HRI safety compared to single-modality or raw-pixel approaches.

• Safety: By detecting internal mechanical failures (via torque sensors) alongside visual anomalies, robots can react to failures before they result in physical damage or injury.
• Efficiency: Operating in a semantic feature space reduces the computational burden, making real-time deployment more feasible.
• Future Work: The authors plan to expand the dataset with more diverse tasks, improve the Scene Graph generation (to make that modality useful), and extend the framework to perform detailed failure classification rather than just binary detection.

In summary, MADRI proves that a robot's "self-awareness" (internal sensors) combined with its "visual understanding" (semantic features) creates a much more robust safety system than vision alone.