⚡ electrical engineering

HoRAMA: Holistic Reconstruction with Automated Material Assignment for Ray Tracing using NYURay

This paper introduces HoRAMA, an automated framework that rapidly generates ray-tracing-compatible 3D digital twins from smartphone-captured video by integrating dense point cloud generation with vision-language material assignment, achieving wireless channel prediction accuracy comparable to manual models while reducing reconstruction time from months to hours.

Original authors: Mingjun Ying, Guanyue Qian, Xinquan Wang, Peijie Ma, Dipankar Shakya, Theodore S. Rappaport

Published 2026-02-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mingjun Ying, Guanyue Qian, Xinquan Wang, Peijie Ma, Dipankar Shakya, Theodore S. Rappaport

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict how a radio signal (like your Wi-Fi or 5G) will bounce around inside a giant, cluttered factory. To do this accurately, you need a perfect 3D map of the factory, including every wall, table, and machine, and you need to know exactly what those things are made of (wood, metal, glass, concrete). Why? Because a radio wave bounces off metal differently than it passes through wood.

The Problem: The "Slow and Expensive" Map
Traditionally, making this map was like hiring a team of architects and surveyors to spend two months inside the factory. They would measure every inch with laser tools, draw every object in 3D software by hand, and manually label every surface as "metal" or "wood." It was incredibly accurate, but it was slow, expensive, and couldn't scale. You couldn't do this for every building in a city.

The Solution: HoRAMA (The "Smart Phone Tour")
This paper introduces HoRAMA (Holistic Reconstruction with Automated Material Assignment). Think of HoRAMA as a "smart phone tour" that builds the map for you in a fraction of the time.

Here is how it works, using a simple analogy:

1. The "Eyes" (Taking the Video)

Instead of a surveyor with a laser, you just walk through the factory with a standard smartphone (like an iPhone) and record a video. You pan around, showing the walls, the machines, and the furniture.

The Magic: The system uses a super-smart AI (called MASt3R-SLAM) that watches the video and instantly figures out the 3D shape of everything, creating a "point cloud" (a digital cloud of dots representing the room). It's like the phone's brain reconstructing the room's geometry just by watching you walk.

2. The "Brain" (Figuring Out What Things Are)

This is the real breakthrough. Usually, computers can see shapes but don't know what the shapes are made of.

The Magic: HoRAMA uses a Vision-Language Model (VLM)—basically an AI that is really good at looking at pictures and reading text.
- It takes the 3D map and cuts out little "snapshots" of every object (a table, a wall, a chair).
- It asks the AI: "What is this made of?"
- The AI looks at the texture in the video and says, "That looks like concrete," or "That looks like polished metal."
- It does this for every single object and uses a "majority vote" (looking at the object from many angles) to make sure it's right.

3. The "Builder" (Making the Final Map)

Once the AI knows the shape and the material, it automatically builds a clean, watertight 3D model that a radio simulation computer can read. It assigns the correct "physics rules" to every surface (e.g., "Metal reflects 90% of the signal," "Glass lets 50% through").

The Results: Speed vs. Accuracy

The researchers tested this in a 700-square-meter factory (about the size of a large gym).

The Old Way: Took 2 months of human work.
The HoRAMA Way: Took 16 hours of computer processing.
The Accuracy: When they simulated radio signals, the HoRAMA map was almost identical to the human-made map.
- The human map had an error of 2.18 dB (a measure of prediction accuracy).
- The AI map had an error of 2.28 dB.
- Translation: The AI was only slightly less accurate, but it was 30 times faster.

Why Does This Matter?

Imagine trying to design a 5G or 6G network for a whole city, or even a whole country. You can't wait two months to build a map for every single building.

Digital Twins: HoRAMA allows us to create "Digital Twins" (virtual copies) of real-world places instantly.
Real-Time Planning: Engineers can walk into a new building, film it with a phone, and immediately know where to put cell towers or how to beam signals to avoid dead zones.
Future Tech: This paves the way for self-driving cars and robots to understand their environment and communicate with each other in real-time, without needing a pre-built map.

In a Nutshell:
HoRAMA turns a simple smartphone video into a high-tech, physics-ready 3D map of a building, automating the boring, slow parts of the job so we can build the wireless networks of the future much faster.

1. Problem Statement

Next-generation wireless networks (5G/6G) operating in upper mid-band and millimeter-wave (mmWave) frequencies require highly accurate, site-specific channel propagation predictions. Ray Tracing (RT) is the standard deterministic method for these predictions, but its accuracy relies heavily on two factors:

Geometric Fidelity: The 3D environment model must be watertight, manifold, and spatially precise to correctly calculate ray intersections, reflections, and diffractions.
Material Characterization: Every surface must be assigned accurate electromagnetic (EM) properties (relative permittivity $\varepsilon_r$ and conductivity $\sigma$ ) to calculate Fresnel reflection and transmission coefficients.

Current Bottlenecks:

Manual Reconstruction: Creating accurate 3D models with material labels typically requires weeks of expert effort (using tools like Blender, laser rangefinders, and datasheets). This is not scalable for large environments (e.g., factories) or rapid network planning.
Traditional Vision-Based Methods: Existing automated 3D reconstruction techniques often produce geometrically defective meshes (non-manifold, noisy) and lack the ability to automatically assign EM-compatible material properties, rendering them unsuitable for wireless RT.
Neural Field Methods: Approaches like NeRF or Gaussian Splatting bypass explicit modeling but require site-specific measurements and retraining for every new environment, preventing predictions for unmeasured sites.

2. Methodology: The HoRAMA Framework

The paper proposes HoRAMA (Holistic Reconstruction with Automated Material Assignment), a pipeline that converts standard RGB video (captured via smartphone) into an RT-ready 3D model with automated material labels. The workflow consists of four main stages:

A. Input and Point Cloud Generation

Input: RGB video captured by a commodity camera (e.g., iPhone 16 Pro) at 60 fps.
Algorithm: Uses MASt3R-SLAM, a real-time monocular dense SLAM system. It generates globally consistent camera poses and dense 3D point clouds with per-pixel confidence scores without requiring depth sensors or calibrated camera intrinsics.
Optimization: Applies loop closure and global optimization to reduce trajectory drift.

B. Outlier Filtering and Cleaning

Raw point clouds contain noise from transient objects, specular reflections, and matching errors.
Filtering Strategy:
- Robust Huber Weighting: Downweights outliers based on reprojection error.
- Geometric Distance Filtering: Removes points with reprojection errors > 0.1m.
- Size Filtering: Eliminates small clusters (<100 points) to remove noise while retaining 85–90% of valid geometry.

C. Semantic Segmentation and Material Classification

This is the core innovation for "Automated Material Assignment":

Segmentation: Uses PTv3 (a deep learning model trained on the S3DIS dataset) to classify point cloud points into semantic categories (e.g., wall, floor, table, chair).
Instance Grouping: Euclidean clustering groups adjacent points of the same label into distinct object instances.
VLM-Based Classification: For each object instance, the system identifies corresponding video frames and crops the relevant RGB regions.
- Model: Uses Qwen3-VL (a Vision-Language Model) to analyze the cropped images and classify the material (e.g., concrete, wood, metal, glass) based on ITU-R P.2040 categories.
- Robustness: Employs majority voting across all frames where the object is visible to handle occlusions and lighting variations.
Mapping: The classified material labels are mapped to frequency-dependent complex permittivity and conductivity values from the ITU-R P.2040 database.

D. Surface Reconstruction and Smoothing

Planar Objects (Walls/Floors): Uses PLANA3R to extract planar primitives. Adjacent primitives are merged and fitted to plane equations to ensure geometric flatness, which is critical for accurate specular reflection in RT.
Non-Planar Objects (Furniture/Machinery): Uses Poisson surface reconstruction to generate watertight manifold meshes.
Output: The final model is exported as a Mitsuba XML file, where every triangle face carries a material label, making it directly compatible with the NYURay engine.

3. Key Contributions

Fully Automated RT-Compatible 3D Reconstruction: HoRAMA eliminates the need for manual CAD modeling and expert material inspection, generating manifold meshes with embedded EM properties directly from RGB video.
VLM-Driven Material Assignment: It introduces a novel method using Vision-Language Models (Qwen3-VL) to classify materials for RT, bridging the gap between visual appearance and electromagnetic properties.
Scalability and Efficiency: The system reduces the reconstruction time for a 700 $m^2$ factory from two months (manual) to 16 hours (automated) using dual NVIDIA RTX 4090 GPUs.
Dual-Band Validation: The method is rigorously validated against field measurements at both 6.75 GHz and 16.95 GHz, demonstrating viability across FR3 and mmWave bands.

4. Experimental Results

The authors validated HoRAMA in the NYU MakerSpace Factory (700 $m^2$ ) using 12 Transmitter-Receiver (TX-RX) locations.

Metric: Root Mean Square Error (RMSE) of matched Multipath Component (MPC) powers in omnidirectional Power Delay Profiles (PDPs).
Performance Comparison:
- Manual Baseline: 2.18 dB RMSE (Combined 6.75 & 16.95 GHz).
- HoRAMA: 2.28 dB RMSE.
- Conclusion: HoRAMA achieves accuracy comparable to the manual baseline (only 0.10 dB difference) while being orders of magnitude faster.
Scenario Breakdown:
- Line-of-Sight (LOS): Both methods perform well (HoRAMA: 1.14 dB vs. Manual: 0.48 dB).
- Non-Line-of-Sight (NLOS): Errors increase due to multipath complexity (HoRAMA: 4.06 dB vs. Manual: 3.46 dB).
Frequency Sensitivity: At 16.95 GHz, HoRAMA's error (2.96 dB) is slightly higher than the manual model (2.20 dB). This is attributed to the smoothing stages in HoRAMA removing fine geometric details (e.g., machine edges) that become significant scatterers at shorter wavelengths ( $\lambda \approx 17.7$ mm).

5. Significance and Impact

Digital Twin Creation: HoRAMA enables the rapid, scalable creation of "wireless digital twins" essential for network planning, infrastructure deployment, and beam management in 5G/6G systems.
Democratization of RT: By using low-cost smartphones and open-source AI models, it removes the barrier of expensive LiDAR scanners and expert CAD modeling, making site-specific RT accessible to a broader range of engineers.
Future Applications: The framework paves the way for real-time edge implementation and can be extended to outdoor urban scenarios using drone-based video capture.

In summary, HoRAMA successfully solves the scalability bottleneck of wireless ray tracing by integrating advanced computer vision (SLAM, Semantic Segmentation) and Generative AI (VLMs) to produce high-fidelity, material-aware 3D models with minimal human intervention.