Technical Summary: OctoSense: Self-Supervised Learning for Multimodal Robot Perception

Problem Statement

Vision-based foundation models trained via self-supervised learning (SSL), such as DINO and SigLIP, have transformed computer vision. However, in real-world robotic settings, no single sensor suffices. Cameras struggle with low light, high dynamic range, and rapid motion; LiDAR provides accurate but sparse depth with poor semantics; and inertial sensors lack spatial context. Furthermore, sensors exhibit different spatiotemporal characteristics, data rates, and failure modes (e.g., weather degradation, packet loss). Existing SSL methods typically focus on visual or textual data, or specific modalities like depth or IMUs, but lack a unified framework to learn fused representations from the diverse suite of sensors a robot typically employs. The paper identifies a critical gap: the absence of large-scale, diverse, multimodal datasets and corresponding SSL methods capable of handling the heterogeneity of robot sensors to build representations resilient to environmental degradation and sensor failure.

Methodology

1. The OctoSense Hardware and Dataset

To address the lack of data, the authors developed OctoSense, an open-source hardware platform and dataset.

• Hardware: The platform integrates eight sensors: stereo RGB cameras, event cameras, a thermal camera, a 64-line LiDAR, an IMU, an RTK-corrected GPS, and proprioception data (CAN bus from a car and joint angles from a quadruped). The system features custom time-synchronization circuits (using PPS signals and Kalman smoothing) and mechanical designs to mitigate vibration and thermal issues.
• Dataset: The release contains 59 hours of time-synchronized data covering 2,474 km across urban, suburban, and rural environments. It includes diverse conditions (sunrise, day, sunset, night) and scenarios with degraded sensors (sunflares, dropped packets). Ground truth for odometry, depth, optical flow, and semantic segmentation is generated using LiDAR-inertial odometry (RKO-LIO) and pseudo-labeling.

2. Multi-Modal Masked Autoencoder (MAE)

The core algorithmic contribution is a late-fusion masked autoencoder designed to handle heterogeneous sensor inputs.

• Input Representation:
  • RGB: Undistorted and rectified images converted to ViT-style patches.
  • Event Cameras: Event streams processed through a bank of leaky integrators to create multi-channel, image-like representations.
  • LiDAR: Point clouds converted to range images (64 × 512) after deskewing.
  • IMU: Accelerometer and gyroscope time series tokenized via cross-attention pooling over 1.6-second windows.
• Tokenization: The authors employ modality-specific tokenizers using Finite Scalar Quantization (FSQ). Instead of minimizing pixel-wise MSE (which fails across modalities with different statistics), the model minimizes the discrepancy between predicted and ground-truth tokens. This allows the model to learn a shared discrete latent space for diverse inputs.
• Architecture (Late-Fusion):
  • Spatial Encoders: Each modality is processed independently by a spatial transformer encoder at each time step.
  • Temporal Encoders: A "tubelet" attention mechanism processes temporal sequences for each modality independently, aggregating information over a 1.4-second window (8 frames).
  • Cross-Modal Encoder: A final transformer layer fuses the tokens from all modalities to create a shared representation.
  • Masking Strategy: The model uses a spatiotemporal tube masking strategy where entire tubes (a patch across time) are masked. The masking ratio is sampled from a Dirichlet distribution to allow for varying degrees of missing modalities during training.
• Inference Optimization: The architecture caches modality-specific features, processing only new measurements at inference time. This "streaming" capability significantly reduces latency compared to early-fusion approaches.

3. Downstream Probes

After pretraining, the frozen encoder is used to train task-specific probes:

• Dense Prediction: A Dense Prediction Transformer (DPT) head predicts depth, optical flow, and semantic segmentation using features from multiple encoder layers.
• Ego-Motion: Independent cross-attention modules predict linear/angular velocity, relative translation/rotation, speed, and steering angle.

Key Contributions

1. OctoSense Dataset: A large-scale (59 hours), open-source multimodal dataset featuring time-synchronized data from stereo RGB, event cameras, thermal, LiDAR, IMU, GPS, and proprioception, covering diverse environments and degraded sensor conditions.
2. Modality-Specific Tokenization: A framework using FSQ and modality-specific tokenizers to handle the distinct spatiotemporal statistics of different sensors within a unified SSL framework.
3. Late-Fusion MAE Architecture: A novel architecture that processes modalities independently through spatial and temporal encoders before fusing them, achieving high performance with significantly lower latency than early-fusion counterparts.
4. Robustness to Degradation: Demonstration that the multimodal approach outperforms vision-only baselines, particularly in nighttime and sensor-degraded scenarios.

Results

The authors evaluated their approach against existing vision-only foundation models (DINOv2, DINOv3, SigLIP 2, Perception Encoder, V-JEPA 2.1) and several baselines (RGB-only MAE, early-fusion MAE) on five tasks: depth estimation, optical flow, semantic segmentation, and ego-motion (translation, rotation, steering).

• Performance Gains:
  • Depth: The late-fusion MAE reduced depth RMSE by 1.65 m compared to the best existing method (V-JEPA 2.1) during daytime, with even larger improvements at night.
  • Optical Flow: Improved end-point error by 7.16 pixels (day) and 8.51 pixels (night) over V-JEPA 2.1.
  • Ego-Motion: The model achieved near-perfect prediction for translation (0.06 m error) and rotation (0.23° error), significantly outperforming RGB-only models which struggle without inertial or geometric cues.
  • Segmentation: Performance was on par with the best existing models (0.411 mIoU vs. 0.410), though the authors note this is likely limited by the quality of the pseudo-labels generated by a DINOv2-based model.
• Robustness: In degraded sensor scenarios, the late-fusion MAE maintained low error rates, whereas vision-only models degraded significantly.
• Ablation Studies:
  • LiDAR: Removing LiDAR caused a sharp increase in depth and flow errors, confirming its role as the dominant sensor for geometric tasks.
  • RGB: Removing RGB primarily hurt segmentation performance.
  • Event/IMU: These contributed marginally to the specific tasks evaluated (likely due to the 5 Hz evaluation rate being far below their native resolution), but the authors note their potential for high-speed tracking.
• Efficiency: The late-fusion architecture achieved a latency of 6.68 ms on an NVIDIA 5090 and 112 ms on an Orin NX, roughly 60% faster than an equivalent early-fusion architecture, making it suitable for real-time robotics.
• Generalization: The model trained on OctoSense showed competitive performance on the M3ED dataset (a different sensor suite), demonstrating zero-shot generalization capabilities.

Significance and Claims

The paper claims that multi-sensor fusion robust to environmental degradation is the defining enabling technology for field robotics, a point echoed by the Global Robotics Technology Roadmap 2025–2035. The authors argue that their work addresses the gap in SSL methods for typical robot sensors by:

1. Providing a large-scale, diverse dataset that includes event cameras and proprioception, which are often missing from driving datasets.
2. Demonstrating that a late-fusion MAE can effectively learn fused representations from heterogeneous sensors, outperforming vision-only foundation models in geometric and motion estimation tasks.
3. Showing that LiDAR is critical for robust ego-motion and depth estimation, while RGB is essential for semantic tasks, and that the proposed architecture can leverage these complementary strengths.

The authors modestly acknowledge that their dataset is orders of magnitude smaller than those used to train industry-scale foundation models (like DINO or V-JEPA) and that their approach is a research prototype. They posit that scaling the data and network size, potentially with longer temporal windows and recurrent architectures, is necessary to build a true "foundation" model for multimodal robot perception. They also note that the marginal contribution of event cameras and IMUs in their current evaluation is likely due to the low evaluation frequency (5 Hz) relative to the sensors' native capabilities, suggesting future work should target high-speed, low-latency maneuvers.