📄 other

Self-contained intraoperative surface acquisition on a mixed-reality headset: a comparison of sensor-based and learning-based reconstruction

This study demonstrates that a commercial mixed-reality headset can acquire intraoperative organ surfaces using only onboard sensors, finding that while its time-of-flight depth sensor generally offers superior accuracy on opaque tissues, learning-based RGB reconstruction provides a more robust alternative for translucent surfaces where active depth sensing fails.

Original authors: Bowen Xiang, Michael I. Miga

Published 2026-07-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bowen Xiang, Michael I. Miga

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect 3D map of a squishy, shifting object, like a giant, living jellyfish, while you are standing right next to it. In the world of surgery, doctors need to do exactly this with human organs. Before an operation, they have a static 3D model from a CT scan, but once the surgery starts, organs like the liver move, squish, and change shape. To guide the surgeon safely, they need to update that digital map in real-time to match the actual, wiggly organ. This is called "deformable registration."

The tricky part is figuring out the shape of the organ right now, without poking it or using a bunch of extra cameras that clutter the operating room. Enter "Mixed Reality" (MR) headsets. Think of these as high-tech glasses that let you see digital holograms floating in the real world. Some of these glasses have built-in "eyes" that can measure distance, kind of like a bat using echolocation but with light. However, these built-in eyes sometimes get confused by shiny or see-through surfaces, like wet skin or gelatin. So, scientists are asking: Can these glasses measure the organ's shape well enough on their own, or do they need a backup plan using just the regular video camera and some smart computer guessing?

This paper dives into that exact question using a Magic Leap 2 headset. The researchers wanted to see if the headset's built-in distance sensor (which shoots out invisible light pulses) could create a perfect 3D map of an organ, or if a "learning-based" method (where the computer looks at regular photos and guesses the depth) would be better. They tested this on fake organs made of different materials: some that were solid and matte (like a white plaster cast), some that were see-through and squishy (like red silicone), and even a real pig liver.

The team found that there isn't one single "best" way; it depends entirely on what the organ looks like. When the surface was solid and opaque (like the white plaster), the headset's built-in distance sensor was the clear winner, creating a map that was accurate to within about 2 to 3 millimeters. It was like using a tape measure: direct and reliable. However, when the surface was translucent or tricky (like the red silicone or a real liver), the built-in sensor got confused. The light would bounce around inside the material, making the organ look closer than it really was. In these cases, the "learning-based" method, which just analyzed the regular video photos, did a better job, avoiding those optical tricks.

The researchers also tested this on a live, anesthetized pig to see if it would work in a real surgery setting. They found that the headset could indeed capture the liver's surface without any extra equipment. However, they noticed a catch: the photo-based method needed to see the organ from many different angles to work well. If the surgeon could only see the liver from one side (which happens in tight surgical spaces), the photo method struggled, but the built-in sensor still managed to get a decent map.

In short, the paper suggests that a commercial MR headset can act as a self-contained tool for mapping organs during surgery, but the best strategy changes based on the tissue. If the tissue is solid and matte, trust the distance sensor. If the tissue is wet, shiny, or see-through, trust the computer's photo-guessing. The authors conclude that by picking the right method for the right tissue, surgeons could one day ditch the bulky external cameras and rely solely on their smart glasses to navigate the shifting landscape of the human body.

Technical Summary: Self-contained Intraoperative Surface Acquisition on a Mixed-Reality Headset

Problem Statement
Mixed-reality (MR) surgical navigation relies on deformable image-to-physical registration, which requires an accurate, intraoperatively measured organ surface to correct for tissue deformation between preoperative imaging and the surgical field. Current commercial MR headsets lack native capabilities to acquire this surface, forcing reliance on manual stylus digitization (slow, sparse, operator-dependent) or external stereo-camera systems (requiring separate calibration, sterile draping, and fixed vantage points prone to occlusion). Furthermore, prior research suggests that onboard depth sensors in MR devices may be insufficient for high-precision tasks due to biases, particularly on translucent tissues. This work addresses whether a commercial MR headset (Magic Leap 2) can function as a self-contained surface acquisition device using only onboard sensors, and whether a sensor-based approach (Time-of-Flight) or a learning-based approach (RGB-only) yields superior accuracy under varying tissue optical conditions.

Methodology
The study developed and compared two reconstruction paradigms using identical input data from a Magic Leap 2 (ML2) headset, all expressed within a single SLAM-tracked world frame ( $F_{ML2}$ ) to eliminate the need for inter-view registration.

Platform Characterization:
- Hologram Stability: The stability of the ML2's SLAM-tracked world origin was evaluated under four perturbations (walking, rapid head movement, sensor occlusion/re-localization, and dynamic scene updates) using an optically tracked stylus.
- Sensor Calibration: The onboard short-range indirect Time-of-Flight (iToF) sensor and RGB camera were calibrated. Radial distance measurements were converted to camera-frame depth to correct for geometric bias.
Data Acquisition & Segmentation:
- Concurrent RGB and depth streams were captured.
- SAM 2 Integration: The Segment Anything Model 2 (SAM 2) was used offline to generate organ-specific masks from RGB frames, filtering out surrounding instruments and drapes. This ensured both reconstruction methods operated on identical organ pixel definitions.
Reconstruction Paradigms:
- Method 1 (Sensor-Based): Geometry was derived directly from the iToF depth stream. Points were back-projected, filtered for outliers, and fused into a single point cloud in $F_{ML2}$ . A grid-based least-squares surface fit was applied to generate a smooth, watertight mesh.
- Method 2 (Learning-Based): The depth channel was discarded. Geometry was recovered solely from the RGB sweep using Depth Anything 3 (DA3), a multi-view geometric foundation model. The resulting unitless 3D structure was anchored to metric scale using the SLAM-tracked camera centers via a closed-form similarity transform (Umeyama solution).
Evaluation:
- Phantoms: Six objects were tested: two liver shapes cast in opaque white plaster and translucent red silicone, a sunscreen-coated breast phantom, and an ex vivo porcine liver.
- Ground Truth: An optically tracked, pivot-calibrated stylus digitized the surfaces. A continuous ground-truth surface was generated via grid fitting to enable dense point-to-point comparison.
- Metrics: Root Mean Square Error (RMSE) of directed point-to-point distances and Symmetric Average Closest-Point Distance (SACPD).
- In Vivo: A qualitative demonstration was performed on an anesthetized pig with an exposed liver.

Key Results

Platform Stability: The ML2 world-origin drift remained below a median of 2 mm across all perturbations, confirming that views from a single sweep can be aggregated directly without additional registration.
Sensor Accuracy: The iToF sensor achieved approximately 2 mm absolute accuracy over the 0.30–0.70 m working range, with temporal noise significantly lower (~0.36 mm median) than absolute error.
Reconstruction Accuracy (Phantoms):
- Opaque Surfaces (Plaster): The sensor-based method was significantly more accurate (RMSE ~2.6–3.1 mm) than the learning-based method (RMSE ~4.7–5.0 mm).
- Translucent Surfaces (Silicone): The ranking reversed. The learning-based method outperformed the sensor-based method (RMSE 4.3–4.6 mm vs. 4.8–5.3 mm), attributed to subsurface scattering biasing the iToF measurements.
- Complex Tissues: On the sunscreen-coated breast phantom, the sensor-based method retained an advantage (2.9 mm vs. 3.9 mm). On the ex vivo porcine liver, the sensor-based method showed a modest, non-significant advantage (4.2 mm vs. 4.8 mm).
In Vivo Demonstration: The sensor-based reconstruction produced a clean surface (mean residual 3.4 mm) on the live pig liver. Conversely, the learning-based method failed to generate a stable surface due to insufficient multi-view coverage (single-sided surgical access), highlighting a dependency on viewpoint diversity.

Significance and Claims
The paper claims to demonstrate that a commercial MR headset can serve as a self-contained intraoperative surface acquisition device, eliminating the need for external tracking hardware or stylus digitization. The primary contribution is the identification of a tissue-dependent trade-off between reconstruction paradigms:

Sensor-based reconstruction is preferred when active depth returns are reliable (opaque, diffuse, or adequately bright surfaces), offering superior absolute positional accuracy.
Learning-based reconstruction is more robust to optical failure modes inherent to active depth sensing (subsurface scattering, specular saturation, low albedo) but requires adequate multi-view coverage and introduces a small residual positional offset.

The authors conclude that the choice of reconstruction paradigm should be dictated by the optical characteristics of the target tissue and the viewpoint diversity permitted by the surgical field. The work establishes the Magic Leap 2 as a viable host for these pipelines, with the next steps being the integration of real-time segmentation and downstream deformable registration.

Technical Summary: Self-contained Intraoperative Surface Acquisition on a Mixed-Reality Headset

More like this