Technical Summary: OMNIROBOTHOME

Problem Statement

Enabling robots to assist in everyday living environments requires continuous, accurate, real-time knowledge of human location and activity in 3D. However, existing human-robot interaction (HRI) systems often rely on limited sensing (e.g., single-view or onboard sensors) or offline processing, which fails to capture the dynamic, unstructured nature of homes. While multi-camera motion capture systems exist, they typically target offline dataset construction or isolated perception rather than delivering dense, real-time state to robots acting in the loop. Consequently, the specific impact of perception quality—such as real-timeness, granularity, coverage, and forecasting—on the success of HRI tasks remains largely unquantified.

Methodology: OMNIROBOTHOME System

The authors present OMNIROBOTHOME, a room-scale physical AI testbed designed to instrument a furnished home with high-fidelity, real-time perception capabilities integrated directly into the robot control loop.

1. Hardware Configuration

• Environment: A 23.1 m² furnished living area (including kitchen, living room, and storage) designed to resemble a real home with natural clutter and occlusions, rather than a sterile capture studio.
• Sensing Array: 48 hardware-synchronized RGB cameras.
  • 40 wide-area cameras distributed for occlusion-robust, markerless human tracking.
  • 8 cameras forming four stereo pairs positioned over specific workspaces for 6D object pose estimation.
• Actuation: Three manipulators integrated into the space: two Franka Research 3 arms (with Inspire F1 hands) and one xArm (with Allegro-v4 hand).
• Synchronization: All cameras and robots are synchronized via an external hardware trigger, ensuring a unified timeline for perception and control.

2. System Architecture & Perception Pipeline

To handle the massive data throughput (~13.5 GB/s raw), the system employs a distributed edge-computing architecture:

• Edge Nodes: 40 wide-area cameras are partitioned across 10 edge nodes. Each node runs TensorRT-optimized inference (YOLO26 for detection, RTMPose for 2D keypoint extraction) and streams only sparse 2D keypoints and confidence scores to a central host.
• Central Fusion: The host triangulates 3D joints in real-time using a batch-vectorized Direct Linear Transform (DLT) solved with Huber-reweighted IRLS, followed by temporal smoothing (One Euro Filter). This pipeline achieves ~22 FPS for full-body 3D pose estimation.
• Object Perception:
  • Room-scale: Coarse 3D location and extent are recovered via multi-view fusion of 2D detections (YOLOE) and volumetric reconstruction.
  • Workspace-scale: Four stereo pairs provide dense metric depth and 6D pose estimation for manipulation targets using FastFoundationStereo and FoundationPose, running at ~16 Hz.
• Calibration: A three-stage process (intrinsics, extrinsics via ChArUco boards and PnP, and hand-eye calibration) unifies all cameras and robots into a single world frame, achieving lower reprojection error than feature-based SfM methods in sparse-feature areas.

3. Core Capabilities

• Real-Time 3D Perception: Markerless tracking of full-body human pose and 6D object pose with low latency.
• Anticipatory Motion Forecasting: A conditional rectified-flow model learns room-specific motion priors from recorded data to forecast 5-second human futures based on 3-second history. It employs a "Best-of-K" sampling strategy and motion recombination via in-betweening to ensure diverse and plausible predictions.
• Multi-Robot Assistance: Manipulators are aligned to the shared perception stream, allowing any nearby robot to assist based on inferred intent.
• Social Avatar Agent: A Vision-Language Model (VLM) agent observes the human via the best camera, reasons about activity and intent, and engages in conversation or issues high-level manipulation commands.

Key Contributions

1. A New Testbed: The first room-scale residential platform integrating 48 synchronized cameras and multiple manipulators in a unified world frame to deliver real-time, full-body 3D perception directly into the robot control loop.
2. A Practical Recipe: An end-to-end real-time sensing pipeline addressing calibration, synchronization, and distributed processing challenges, enabling reproducibility.
3. A Controlled Study: The first study treating perception quality as an experimental variable across safety, assistance, and social interaction domains.

Experimental Results

The authors evaluated the system across three HRI problems, systematically degrading perception quality (via latency, reduced camera count, or lack of forecasting) to measure the impact on interaction quality.

• Safety in Shared Spaces:
  • A policy using the system's real-time forecasting (1.5s horizon) achieved the lowest collision rate (8.60%) compared to baselines like base proximity (18.66%) or no safety (43.53%).
  • Introducing perception delays (0.5s or 1.0s) significantly increased collision rates, demonstrating that real-timeness is critical for safety.
• Human Motion Forecasting:
  • Models trained with more data (4+ hours) and motion recombination (Best-of-K) showed improved accuracy (lower minADE) and diversity (higher APD).
  • Removing the Best-of-K mechanism sharply reduced diversity and recall, confirming the need for multi-modal supervision.
• Spatial Coverage & Assistance:
  • Pointing Task: In a task where a robot must sort items based on a human's pointing gesture, accuracy dropped from 95% (40 cameras) to 41% (4 cameras), proving that dense coverage is essential for fine-grained intent estimation.
  • Calibration: The proposed ChArUco-based calibration reduced reprojection error mean and standard deviation compared to COLMAP.
• Social Interaction:
  • Mutual Gaze: The social avatar's ability to maintain mutual gaze degraded measurably with increased latency and reduced camera coverage, highlighting the need for low-latency, room-scale sensing for natural engagement.

Significance and Claims

The paper argues that perception quality is the dominant factor governing what human-robot interaction is achievable in real-world home environments. The study demonstrates a consistent trend: interaction quality degrades systematically as perception attributes (real-timeness, granularity, coverage, accuracy, forecasting, or memory) are weakened.

The authors conclude that improving perception is a key lever for developing capable home robots. They posit that the "strong-perception regime" provided by OMNIROBOTHOME yields measurable gains over the limited sensing assumptions of prior work. The paper commits to releasing all code and data to support future research in human-robot collaboration that has previously been gated by infrastructure limitations.

Limitations: The evaluation is currently confined to a single instrumented room with fixed-base manipulators and specific task controllers. Generalization to diverse home layouts, mobile robots, and longer-term interactions remains an open direction for future work.