💻 computer science

Mixed Neural Rendering: Interactive Mixed Reality Capture and Curation for Neural 3D Reconstruction

This paper introduces Mixed Neural Rendering (MiNeR), an interactive Mixed Reality pipeline using HoloLens 2 that integrates real-time capture, in-situ inspection, and promptable segmentation to overcome traditional acquisition limitations and enable robust, object-centric neural 3D reconstruction.

Original authors: Alessio Paolucci, Lorenzo Stacchio, Emanuele Balloni, Pasquale Cascarano, Primo Zingaretti

Published 2026-07-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alessio Paolucci, Lorenzo Stacchio, Emanuele Balloni, Pasquale Cascarano, Primo Zingaretti

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a perfect, photorealistic 3D copy of your favorite toy using a camera. Usually, you'd take a bunch of photos, send them to a supercomputer, and wait hours for it to figure out where you were standing for each shot. If you missed a spot or took a blurry picture, the computer might crash, and you wouldn't know until it was too late. It's like baking a cake and only realizing you forgot the eggs after it's already in the oven.

Enter MiNeR (Mixed Neural Rendering), a new system designed by a team of researchers to fix this "baking disaster" problem. They built a workflow that lets you capture, check, and fix your photos while you are taking them, using a special headset called the HoloLens 2.

The "Magic Glasses" Workflow

Think of the HoloLens 2 not just as a camera, but as a pair of magic glasses that see the world in 3D. When you wear them and take a photo, the glasses instantly know exactly where you are standing and which way you are looking.

In the old way, you'd take photos, go home, and hope the computer could figure out the angles. With MiNeR, as you walk around your object, you see little floating "ghost cameras" in your vision showing exactly where you've taken pictures.

The Problem: If you see a gap where you haven't taken a photo, or a blurry photo floating there, you can delete it right then and there.
The Fix: You can walk to that empty spot and snap a new picture immediately. You are curating your own dataset in real-time, acting as a human editor before the computer even starts working.

The Three Big Experiments

The researchers didn't just build the system; they put it through a rigorous "taste test" with nine different real-world objects (like a water bottle, a backpack, and a fan). They tested three different variables to see what made the best 3D copy:

The "GPS" (Pose Source): Did they use the headset's built-in tracking (Direct) or a powerful offline computer program called COLMAP (SfM) to figure out the angles?
The "Mask" (Segmentation): Did they use an AI tool (SAM 2) to cut out the background and focus only on the object, or did they let the background clutter stay in?
The "Engine" (Renderer): Did they use the older "NeRF" method or the newer "3D Gaussian Splatting" (3DGS) method to build the 3D model?

What They Found (The Scoreboard)

The results were clear, but with some important caveats.

1. The "Gold Standard" (Best Quality)
When everything went perfectly, the absolute best results came from combining three things:

Using the COLMAP computer program for angles.
Using the SAM 2 AI to mask out the background.
Using the 3D Gaussian Splatting (3DGS) engine.

This combo produced the highest quality images, scoring a PSNR of 31.78, an SSIM of 0.964, and an LPIPS of 0.050. (Think of PSNR as a "clarity score" where higher is better; 31.78 is very sharp).

2. The "Plan B" (Reliability)
Here is the twist: The "Gold Standard" (COLMAP) is fragile. In their tests, it failed to work on 4 out of 9 objects (only succeeding on 55% of the sequences). It's like a high-end sports car that breaks down if the road is a little bumpy.

However, the "Direct" method (using the headset's own tracking) never failed. It worked on 100% of the objects (9 out of 9).

The Trade-off: The Direct method produced lower quality images on its own (PSNR around 13.27 to 13.36 without masking) compared to the Gold Standard. But, it never crashed. The authors suggest this is a vital "fallback" option: if the fancy computer can't figure it out, the headset's own tracking keeps the process alive.

3. The Power of the "Mask"
No matter which method they used, cutting out the background with the AI mask made a huge difference.

For the 3DGS engine, masking improved the quality score by +12.37 points when using COLMAP, and by +10.04 points when using the Direct headset poses.
Basically, telling the computer "ignore the messy room, look only at the chair" stopped it from getting confused by background clutter.

4. The Winner of the Engines
The newer 3D Gaussian Splatting (3DGS) method consistently beat the older NeRF method in almost every test. It was faster, handled the "Direct" poses better, and produced sharper results.

The Bottom Line

The paper suggests that MiNeR is a game-changer for making 3D models from casual photos because it puts the human in the loop. It doesn't promise that the headset's tracking is perfect (it admits the headset's angles can be "drifty" or less accurate than the super-computer's). Instead, it offers a safety net: you can get the highest possible quality if the computer works, but you can always get a usable result if you use the headset's own tracking and AI masking.

It's not a magic wand that solves every problem instantly; the authors admit their test was small (only nine objects in a room) and that they didn't measure the exact geometric accuracy with a ruler. But for now, it suggests that mixing human curation with AI masking and the right 3D engine is the most reliable way to turn a messy room into a clean, reusable 3D asset.

Technical Summary: Mixed Neural Rendering (MiNeR)

Problem Statement
Neural rendering (NR) methods, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), have demonstrated the ability to generate photorealistic novel views and 3D reconstructions from casually captured images. However, their practical reliability is heavily contingent on acquisition quality, specifically sufficient multi-view coverage and accurate camera poses. Traditionally, these requirements are met through offline Structure-from-Motion (SfM) pipelines (e.g., COLMAP). This approach introduces significant limitations: it is computationally expensive, fragile to imperfect captures (e.g., low texture, motion blur, limited overlap), and often fails only after the capture session is complete. Consequently, users lack the ability to inspect data quality, identify coverage gaps, or curate datasets during the acquisition process, leading to wasted effort and failed reconstructions.

Methodology: The MiNeR Pipeline
The authors introduce Mixed Neural Rendering (MiNeR), a Mixed Reality (MR) capture-to-reconstruction pipeline designed to integrate acquisition, inspection, curation, and validation into a single interactive workflow. The system utilizes a Microsoft HoloLens 2 as both the sensing device and the immersive interface.

The pipeline operates through a client-server architecture:

MR Client (Acquisition & Curation): The user captures RGB images while the headset tracks 6-DoF poses via visual-inertial SLAM. Unlike traditional workflows, MiNeR provides real-time, in-situ feedback by rendering captured viewpoints as "spatial camera proxies" anchored in the physical environment. This allows users to visualize coverage distribution, identify missing angles, and interactively remove blurred or redundant frames. Users can also perform targeted recaptures within the same session.
Promptable Segmentation: To support object-centric reconstruction, the system integrates SAM 2 (Segment Anything Model 2). Users provide a single annotation on a representative frame, and the model propagates foreground masks across the entire sequence, isolating the subject of interest from background clutter.
Server (Processing & Reconstruction): The server ingests the curated dataset and supports a controlled, factorized experimental design ( $2 \times 2 \times 2$ $2 \times 2 \times 2$ ) varying three binary factors:
- Masking: Off (raw images) vs. On (SAM 2 foreground masks).
- Pose Source: COLMAP-based SfM poses vs. Direct headset SLAM poses.
- Neural Representation: NeRF (NeRFacto) vs. 3D Gaussian Splatting (Splatfacto).
  The system includes a coordinate transformation layer to align Unity-native headset poses with the OpenGL/Blender conventions used by Nerfstudio.

Key Contributions

MiNeR System: An MR-guided pipeline that transforms situated visual fragments into spatially grounded neural assets, keeping the operator explicitly in the loop during data acquisition.
Interactive Workflow: A unified interface for capturing, inspecting, and curating datasets in 3D space, enabling the removal and replacement of views before reconstruction.
Pose Interoperability: A framework allowing the same captured session to be processed with either high-fidelity SfM poses or SfM-free direct sensor poses, facilitating a direct comparison of reconstruction pathways.
Promptable Segmentation Integration: The incorporation of single-shot, multi-view segmentation to enable foreground-focused training and controlled masked/unmasked comparisons.
Technical Evaluation: A controlled study on a dataset of nine real-world objects, quantifying the trade-offs between pose accuracy, segmentation, and rendering paradigms.

Experimental Results
The authors evaluated the system on a dataset of nine indoor object-centric sequences (e.g., furniture, electronics, personal items) captured with the HoloLens 2.

Pose Source Impact: When SfM (COLMAP) succeeds, it yields the highest reconstruction fidelity. However, COLMAP failed to converge on 44% of the sequences (5/9 successful), whereas the direct headset pose pathway succeeded on 100% of the sequences. Direct poses serve as a robust, SfM-free fallback, though they exhibit lower fidelity due to SLAM drift and scale inconsistencies.
Segmentation Impact: The use of SAM 2 masking significantly improved reconstruction quality across all configurations. For 3DGS with COLMAP poses, masking increased PSNR by +12.37 dB (from 19.41 to 31.78 dB). Even with direct poses, masking provided substantial gains (e.g., +10.04 dB for 3DGS), indicating that foreground isolation mitigates background-induced optimization noise.
Rendering Paradigm: 3D Gaussian Splatting (3DGS) consistently outperformed NeRF in terms of PSNR, SSIM, and LPIPS across both pose sources and masking conditions. The best overall configuration (COLMAP + SAM + 3DGS) achieved 31.78 dB PSNR, 0.964 SSIM, and 0.050 LPIPS.
Efficiency: Direct pose configurations reduced end-to-end time by bypassing the computationally intensive SfM preprocessing stage.

Significance and Claims
The paper positions MiNeR as a solution to the "capture-reconstruction gap" in XR memory systems. The authors claim that by making capture quality visible and actionable during acquisition, MiNeR transforms casual visual experiences into reusable neural assets.

The study concludes that while classical SfM remains the gold standard for maximum fidelity when conditions are favorable, it is brittle in real-world scenarios. MiNeR offers a complementary strategy: direct headset poses ensure operability when SfM fails, while promptable segmentation and 3DGS maximize quality within those constraints. The system effectively bridges the divide between casual MR acquisition and reconstruction-ready spatial content, enabling users to inspect, validate, and curate data before committing to expensive offline processing.

The authors explicitly note limitations, including the small scale of the dataset (nine objects), the reliance on a single MR device, the lack of external geometric ground truth, and the absence of a user study to measure cognitive load or acquisition efficiency. Future work is proposed to expand the dataset, test diverse devices, and include quantitative geometric error metrics.

The "Magic Glasses" Workflow

The Three Big Experiments

What They Found (The Scoreboard)

The Bottom Line

Technical Summary: Mixed Neural Rendering (MiNeR)

More like this