Ego-OSCAR: Egocentric Open source Stereo CAptuRe System

Technical Summary: Ego-OSCAR

Problem Statement
The development of Vision-Language-Action (VLA) models and world models is increasingly bottlenecked by the scarcity of diverse, large-scale, multimodal datasets that capture real-world interactions. While teleoperation offers high-fidelity data, it does not scale. Simulation suffers from the sim-to-real gap, and autonomous robot farms are limited by safety constraints. Egocentric (first-person) data collection offers a scalable alternative, yet existing solutions present a trade-off: consumer datasets (e.g., Ego4D) rely on monocular, rolling-shutter cameras without hardware-synchronized inertial streams, while high-fidelity research platforms (e.g., Project Aria) are closed-source, expensive, and cannot be freely distributed or reproduced by the broader community. Consequently, there is no open, low-cost, head-mounted device capable of capturing synchronized stereo-inertial data at scale.

Methodology
Ego-OSCAR addresses this gap through an open-hardware, low-cost capture system and a corresponding software stack designed for distributed deployment.

• Hardware Design: The device is a head-mounted unit (approx. 280g) built entirely from commercially available components (COTS) and 3D-printed parts, with a Bill of Materials (BoM) under USD 200.

  • Sensors: It utilizes a Dexcin USB stereo camera module featuring two Omnivision global-shutter sensors hardware-synchronized via a single ASIC. This ensures microsecond-level synchronization and eliminates rolling-shutter artifacts during rapid head motion. The system includes a 6-axis IMU (ICM-20948) sampled at 120 Hz.
  • Compute & Control: A Radxa Rock 5C (Rockchip RK3588 SoC) handles hardware-accelerated video encoding (MJPEG to H.264) and storage. A Seeed Studio Xiao ESP32-S3 microcontroller manages user feedback (RGB LED status), watchdog functions (detecting system hangs), and, crucially, time synchronization.
  • Synchronization Mechanism: To bridge the independent clocks of the camera and IMU, the ESP32 captures the camera's Start-of-Exposure (SoE) signal via an interrupt. It records monotonic timestamps for each frame exposure and merges them with IMU data. An offline alignment process uses a visual anchor (a blue LED flashing on the 60th interrupt) to deterministically align the IMU stream with video frame indices, achieving a residual lag of 700 µs.
  • Robustness: The system includes a watchdog that detects SBC hangs (via a 1 Hz heartbeat) and triggers an error state, preventing silent data loss.

• Software Pipeline: The open-source stack includes a hardware-accelerated recording daemon, an IMU sampling service, time-synchronization tooling, and a watchdog firmware. Data is segmented into 5-minute clips to limit loss during power failures and uploaded with integrity verification.

• Dataset Collection: The authors deployed the system across a distributed network of 25 contributors in India over six months, collecting 1,462 sessions totaling approximately 550 hours of stereo video per camera (1,100 camera-hours). The data covers diverse indoor environments, with a focus on kitchen activities but including sewing and tailoring.

Key Contributions

1. Open-Hardware Device: A fully documented, reproducible head-mounted stereo-inertial capture rig with CAD files, wiring diagrams, and assembly instructions.
2. Open-Source Software Stack: A complete pipeline for recording, synchronization, and data management, enabling the community to replicate the capture process.
3. Ego-OSCAR-550h Dataset: A curated dataset of ~550 hours of egocentric stereo video with synchronized IMU data. Unlike raw sensor streams, this release includes:
   • Free-form action captions: ~209,315 segments covering the entire timeline with an open vocabulary.
   • 3D Hand Reconstructions: Per-frame 3D hand poses generated by WiLoR.
   • Per-session Calibration: Individual calibration files for every session rather than nominal factory intrinsics.

Results and Evaluation
The paper evaluates the system across three tiers:

• Sensor Fidelity: Per-session calibration yields reprojection errors below 0.03 pixels. The hardware-synchronized global shutter and IMU allow for successful visual-inertial odometry (VINS-Fusion) in 12/20 held-out short sequences, with a residual visual-inertial lag of 700 µs. The IMU noise floor is characterized as consumer-grade, suitable for gravity alignment and motion classification but not for long-horizon inertial integration without higher-grade swaps.
• Data Utility: Stereo depth estimation (SGBM, RAFT-Stereo) succeeds across the full 126° field of view. Hand detection rates reach 94%, indicating that the head-mounted geometry effectively keeps hands in the field of view for the majority of interactions.
• Deployment Scale: The system achieved a 96% usable-session rate across 1,462 sessions. Failure modes (thermal shutdown, SD card I/O errors, cable strain) were identified and mitigated through hardware revisions (heatsinks, mechanical reinforcement) and software validation.

Significance and Claims
The paper positions Ego-OSCAR not as a replacement for research-grade systems like Project Aria in terms of per-unit fidelity, but as the "cheapest defensible substrate for crowdsourced egocentric capture." Its primary significance lies in lowering the activation energy for teams to collect egocentric data at scale. By providing a fully open, reproducible, and affordable platform, Ego-OSCAR aims to democratize the collection of high-quality, synchronized stereo-inertial data, complementing existing monocular or closed-platform datasets. The authors explicitly state that demonstrating end-to-end policy gains from this data is future work; the current contribution is the infrastructure and dataset required to enable such research.