🧬 biology

Self-Calibrated Hyperspectral Neural Radiance Fields for 3D Reconstruction of Bone and Bone Analogues

The paper introduces BoNeRF-HS, a self-calibrated hyperspectral neural radiance field framework that enables high-quality, pose-free 3D reconstruction of bone and bone analogues by jointly optimizing camera intrinsics and spectral radiance while leveraging a gated spectral adapter to capture wavelength-dependent tissue details.

Original authors: Neetu Sigger, Tuan Thanh Nguyen, Sadaf Ashraf, Gianluca Tozzi

Published 2026-10-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Neetu Sigger, Tuan Thanh Nguyen, Sadaf Ashraf, Gianluca Tozzi

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine trying to take a perfect three-dimensional photograph of a piece of bone. In a standard camera, you rely on the texture of the surface—the cracks, the pores, the roughness—to tell the computer where one point ends and another begins. But bone is often smooth, shiny, and strangely uniform, making it difficult for traditional cameras to figure out their own position or the shape of the object they are looking at. This is a major hurdle for doctors and scientists who need to see bone in three dimensions without touching it, especially when they also want to know what the bone is made of at a chemical level. To solve this, researchers have developed a new way to build 3D models that does not need the camera to know where it is standing beforehand. Instead, the computer learns the camera's position and the object's shape at the same time, using a special kind of light that reveals the hidden chemical makeup of the tissue.

The challenge of seeing bone clearly is not just about shape; it is about composition. While a regular camera sees only three colors, a hyperspectral camera sees hundreds of narrow bands of light, stretching from the visible spectrum into the near-infrared. This allows the camera to detect subtle differences in how light bounces off mineralized tissue, revealing information about the bone's health, density, and chemical structure. However, combining this rich chemical data with a 3D model has been difficult because the methods used to build 3D shapes usually fail on bone. Standard tools rely on finding matching patterns in the images to guess the camera's movement, but bone surfaces are too smooth and reflective for these tools to work reliably. When the camera's position is guessed incorrectly, the resulting 3D model is blurry or distorted, and the chemical data gets smeared across the wrong parts of the surface.

To overcome this, a team of researchers from the University of Greenwich and Medway School of Pharmacy developed a new system called BoNeRF-HS. This system is a type of artificial intelligence that learns to create a 3D map of an object directly from a series of images taken from different angles, without needing any pre-calculated information about where the camera was. The researchers tested this system on three specific types of specimens: a real mouse bone, a synthetic material designed to mimic the porous, spongy structure of trabecular bone, and another synthetic material mimicking the dense, hard outer layer of cortical bone. They captured 45 different views of each specimen, with each view containing 120 distinct bands of light ranging from 400 to 1000 nanometers.

The core of this new method is a process where the computer figures out the camera's internal settings and its position for every single photo while simultaneously building the 3D shape. In previous attempts, scientists had to use separate software to estimate the camera's position first, a step that often failed on bone. BoNeRF-HS removes that step entirely. It treats the camera's position and the object's 3D density as things that can be learned and improved together. To make the chemical data even more accurate, the system includes a special component that looks at how neighboring bands of light relate to one another. Since light at one wavelength often behaves similarly to light at the next, this component makes small, intelligent adjustments to the predicted colors, ensuring that the chemical signature of the bone remains sharp and true to life.

When the researchers compared their new system against existing methods, the results were clear. The new approach produced sharper images and more accurate 3D shapes for all three specimens. For the mouse bone, the system achieved a level of detail that previous methods could not match, preserving the fine edges and surface textures that are often lost. On the synthetic bone analogues, the system successfully captured the complex porous structure of the spongy bone and the smooth surface of the dense bone, with significantly less error in the color and chemical data than the older techniques. The system was also tested on soft mouse tissue, where it performed well, though the results showed that the method behaves slightly differently on soft, deformable tissue compared to rigid bone, a distinction the researchers noted for future study.

The significance of this work lies in its ability to create a complete, non-contact view of bone that includes both its physical shape and its chemical composition. This is a crucial step forward for fields like orthopedic surgery, where surgeons might one day use such a system to see exactly where a fracture is and how the bone is healing without exposing the patient to radiation. It also holds promise for forensic science, where understanding the condition of skeletal remains can provide vital clues. By proving that a 3D model can be built accurately from hyperspectral data without needing to know the camera's position in advance, the researchers have opened a practical path toward visualizing bone in a way that was previously impossible, turning a difficult imaging problem into a reliable tool for medical and scientific discovery.

Technical Summary: Self-Calibrated Hyperspectral Neural Radiance Fields for 3D Reconstruction of Bone and Bone Analogues

Problem Statement
Hyperspectral imaging (HSI) offers rich, wavelength-dependent information crucial for assessing mineralized tissue composition, while 3D reconstruction preserves surface morphology. Combining these modalities provides a non-contact approach for bone assessment. However, existing hyperspectral 3D reconstruction methods typically rely on externally estimated camera poses (e.g., via COLMAP). This assumption is often unreliable for bone surfaces due to their low texture, specular reflectance, and irregular morphology, which hinder feature matching and pose estimation. Furthermore, standard geometric surface models lack information regarding tissue composition and spectral responses. While Neural Radiance Fields (NeRFs) have advanced 3D scene representation, most hyperspectral NeRF variants assume known camera parameters, limiting their utility in scenarios where external calibration is infeasible.

Methodology: BoNeRF-HS
The authors propose BoNeRF-HS, a self-calibrated hyperspectral neural radiance field framework designed to jointly optimize camera intrinsics, per-view camera poses, volume density, and hyperspectral radiance directly from multi-view HSI data, eliminating the need for pre-composed poses (e.g., COLMAP).

The framework consists of the following key components:

Self-Calibrated Camera Parameters: Inspired by NeRF--, camera intrinsics (focal lengths $f_x, f_y$ ) and extrinsics (rotation $R_i$ and translation $t_i$ ) are treated as learnable parameters optimized alongside the radiance field. This removes the dependency on external calibration.
Hyperspectral NeRF Backbone: The model utilizes a Multi-Layer Perceptron (MLP) that maps 3D spatial locations and viewing directions to a latent feature vector. Unlike standard NeRFs that predict RGB, this backbone predicts wavelength-dependent spectral responses across $B$ bands (400–1000 nm).
Gated Spectral Adapter Head: To exploit correlations between neighboring spectral bands, the authors introduce a gated spectral adapter. This module learns a controlled, wavelength-dependent residual correction ( $\Delta S$ $Δ S$ ) to the initial direct spectral prediction ( $S_{dir}$ $S_{d i r}$ ).
- A direct spectral head predicts the coarse hyperspectral radiance.
- A spectral adapter predicts a residual correction.
- A gate branch predicts a confidence value ( $g \in (0,1)$ ) to control the magnitude of the correction.
- The final spectrum is computed as $S_{final} = S_{dir} + g \cdot \Delta S$ . This allows the model to refine spectral responses while preserving the global spectral structure.
Training Objective: The model is trained using the Spectral Angle Mapper (SAM) loss to preserve spectral-shape similarity between predicted and ground-truth spectra, alongside standard rendering losses.

Dataset and Experimental Setup
The study utilizes a newly acquired multi-view hyperspectral dataset comprising:

Specimens: One mouse bone sample, one trabecular bone analogue, and one cortical bone analogue.
Acquisition: 45 views per specimen captured from 0° to 360° using a NIREOS HERA VNIR camera (400–1000 nm, 120 spectral bands).
Preprocessing: Includes reflectance calibration using dark and white references, and foreground masking to remove stationary background artifacts.

Key Results
The framework was evaluated against existing self-calibrated and pose-dependent methods (NeRF--, SiNeRF, NoPe-NeRF) using Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Root Mean Squared Error (RMSE).

Quantitative Performance: BoNeRF-HS achieved the best PSNR and SSIM scores across all three sample types (mouse bone, trabecular analogue, cortical analogue). For the mouse bone, it achieved a PSNR of 29.94 and SSIM of 0.9254, outperforming the next best method (NeRF-- with 27.87 PSNR). It also consistently achieved the lowest RMSE, indicating superior spectral accuracy.
Qualitative Performance: Visual comparisons at specific spectral bands (e.g., band 50) demonstrated that BoNeRF-HS produced sharper rendered views and better preservation of fine surface details (edges, textures, and porous structures) compared to baselines. Error heatmaps showed reduced residuals around object boundaries and local structures.
Spectral Fidelity: Pixel-wise spectral curves showed that BoNeRF-HS more closely followed ground truth spectra, particularly in near-infrared bands where noise is typically higher.
Ablation Study: The gated spectral adapter significantly improved performance. The "learned gate" variant improved PSNR by nearly 3 dB over the direct spectral head alone, confirming that adaptive control of residual correction is superior to fixed or un-gated corrections.
Soft Tissue Analysis: When tested on a mouse tissue sample, BoNeRF-HS achieved the best PSNR and RMSE but slightly lower SSIM compared to NoPe-NeRF. The authors attribute this to the different surface characteristics of soft tissue (deformable, low contrast) versus rigid bone, suggesting SSIM may weigh structural contrast differently than pixel-wise intensity errors in non-rigid scenarios.

Significance and Claims
The paper claims that BoNeRF-HS establishes a practical path toward pose-free, non-contact 3D hyperspectral characterization of bone. By removing the reliance on COLMAP-based poses, the method addresses a critical bottleneck in imaging low-texture, specular surfaces like bone.

The authors highlight potential applications in:

Intraoperative Visualization: Supporting image-guided orthopedic procedures where both surface structure and tissue-specific spectral responses are vital.
Forensic Skeletal Analysis: Providing a non-contact method for documenting skeletal samples while preserving morphology and spectral data.
Biomaterial Evaluation: Assessing bone analogues and synthetic materials.

The authors maintain a modest stance, noting that the current experiments are limited to a single sample per type and a controlled benchtop setting. They acknowledge that translation to real surgical environments (dynamic, freehand, variable illumination) requires further validation. Additionally, they identify the performance divergence on soft tissue (SSIM vs. PSNR) as an open question requiring further investigation into how the method handles deformable, low-texture tissues compared to rigid mineralized bone.

Technical Summary: Self-Calibrated Hyperspectral Neural Radiance Fields for 3D Reconstruction of Bone and Bone Analogues

More like this