Technical Summary: Edge-XReal

Problem Statement

Extended Reality (XR) applications, particularly Mixed Reality (MR), require high-fidelity 3D Holistic Scene Understanding (HSU) to reconstruct physical environments, including object meshes, layouts, and camera poses. While Deep Learning (DL) models have achieved significant accuracy in HSU (e.g., Implicit3D, Total3D, 3D-Scene-Former), their computational demands exceed the onboard capabilities of current standalone Head-Mounted Displays (HMDs). These models often contain tens of millions of parameters and require substantial Video RAM (VRAM), which is unavailable on low-power edge devices. Furthermore, commercial VR headsets lack native APIs for major DL frameworks (PyTorch, TensorFlow), making the direct deployment of state-of-the-art HSU pipelines on edge devices infeasible. This limitation hinders the development of immersive, real-time XR experiences where dynamic scene understanding is critical.

Methodology

To address this hardware bottleneck, the authors propose Edge-XReal, a distributed client-server architecture designed to offload intensive DL inference tasks from the edge device to a remote server while retaining data capture and visualization on the HMD.

System Architecture

• Client Side (Edge Device): Implemented as a Unity application running on a Meta Quest 3. The client captures a single RGB image and the camera intrinsic matrix (calibrated by the authors for the Quest 3) and transmits this data to the server via a network connection. Upon receiving the processed data, the client visualizes the 3D scene reconstruction in the XR environment.
• Server Side: A centralized server (equipped with an Intel i7-3770 CPU and NVIDIA GTX 1070 GPU) hosts the heavy computational models. The server workflow involves:
  1. Receiving the image and intrinsic matrix.
  2. Performing 2D object detection to generate bounding boxes.
  3. Executing the HSU pipeline (e.g., 3D-Scene-Former, Implicit3D) to estimate room layout, object poses, and implicit shape representations.
  4. Converting implicit representations into 3D meshes.
  5. Compressing the results (object meshes, poses, layout) into a JSON file, zipping it to minimize network latency, and transmitting it back to the client.

Pipeline Integration

The system is designed to be agnostic to the specific 2D detector and HSU pipeline used. The authors evaluated several combinations:

• 2D Detectors: Faster R-CNN (with ResNet50 and MobileNetV3 backbones) and DETR (with ResNet50).
• HSU Models: Total3D, Implicit3D, and 3D-Scene-Former.
• Data Handling: The pipeline includes a refinement stage where the server processes the 2D detections to generate 3D bounding boxes, refines object placement using scene graphs or transformers, and generates meshes using techniques like Marching Cubes or Deep Marching Tetrahedra (DMTet).

Key Contributions

1. Distributed Architecture for Edge XR: The paper presents a novel framework that enables the use of computationally intensive HSU models on low-power HMDs by decoupling data capture from inference.
2. Comprehensive 2D Detector Evaluation: Unlike previous HSU works that assume 100% accurate 2D ground truth, this study integrates and evaluates 2D object detection as a necessary first step. The authors found that Faster R-CNN with a ResNet50 backbone offers the best balance of accuracy and speed for this pipeline, whereas DETR showed high recall but lower precision due to overlapping detections.
3. Proof of Concept (POC): A functional Unity-based application was developed and tested on a Meta Quest 3, demonstrating the feasibility of real-time 3D scene reconstruction on a consumer-grade HMD via a remote server.
4. Optimization Strategies: The implementation utilizes GZIP compression to reduce network payload size, significantly offsetting transfer times despite the added compression overhead.

Results

The evaluation focused on processing time, energy consumption, and system scalability:

• Performance Improvement: Edge-XReal achieved a 40% to 48% reduction in total processing time compared to a hypothetical "on-demand" local execution model where the algorithm is loaded and run from scratch for every request. This is primarily due to the server keeping models loaded in VRAM, eliminating the model loading time (which ranged from ~1.2s to ~2.6s in local benchmarks).
• Latency Breakdown: In the distributed setup, the HSU model prediction remains the most time-consuming step, followed by networking. The use of compression reduced transfer time by approximately 1 second on the test hardware.
• Concurrency: The system demonstrated stable performance with request intervals of at least 2 seconds, theoretically supporting up to 30 concurrent users if requests are spread out. However, error rates increased noticeably when four or more requests were issued simultaneously, indicating a need for better queueing strategies for burst traffic.
• Accuracy: The 2D detector evaluation revealed that while Faster R-CNN (ResNet50) is the most reliable, incorrect 2D detections still drastically affect HSU outputs, highlighting a resilience limitation in current HSU algorithms.

Significance and Claims

The paper claims that Edge-XReal provides a future-proof and flexible solution for bringing complex DL techniques to the edge. By abstracting the underlying 2D detector and HSU pipeline, the architecture allows developers to adapt to new models without redesigning the entire system.

The authors emphasize that this approach makes high-quality 3D scene understanding accessible on devices that previously could not support it, enabling new immersive XR experiences. The work serves as a foundational step for researchers and industry leaders to integrate advanced DL models into XR, shifting the computational burden to the cloud while maintaining a responsive user experience on the device. The authors note that while the current POC proves the concept, future work involves refining queueing strategies for high-concurrency scenarios and improving the resilience of HSU models to imperfect 2D detections.