HS-3D-NeRF: 3D Surface and Hyperspectral Reconstruction From Stationary Hyperspectral Images Using Multi-Channel NeRFs

This paper introduces HSI-SC-NeRF, a stationary-camera multi-channel NeRF framework that enables high-throughput, accurate 3D geometric and hyperspectral reconstruction of agricultural produce by rotating objects in a custom imaging chamber and employing a two-stage training protocol to integrate multi-view data for automated postharvest inspection.

The Gist

Imagine you are a detective trying to solve a mystery about a piece of fruit. You want to know two things: what it looks like (its shape, size, and bumps) and what's happening inside it (is it ripe? is it bruised? does it have enough water?).

Usually, taking a 3D photo of an object requires moving a camera around it, like a photographer circling a model. But for a hyperspectral camera (a super-powerful camera that sees hundreds of colors, not just the three our eyes see), moving the camera is a nightmare. It's heavy, expensive, and hard to keep steady.

This paper introduces a clever solution called HS-3D-NeRF. Here is how it works, explained simply:

1. The Setup: The "Spinning Top" Trick

Instead of moving the camera, they keep the camera perfectly still and spin the fruit on a special turntable.

• The Analogy: Think of a potter's wheel. The potter (the camera) stands still, but the clay (the fruit) spins around.
• The Room: They built a special room lined with Teflon (the same non-stick material used in frying pans). Why? Because Teflon bounces light around evenly, like a giant, soft, white fog. This ensures the fruit is lit perfectly from every angle, so the camera doesn't get confused by shadows or glare.

2. The Camera: The "Super-Eye"

They use a special camera that doesn't just see Red, Green, and Blue (like your phone). It sees 204 different colors (bands) ranging from visible light to near-infrared.

• The Magic: This allows the camera to "see" things invisible to us, like how much water is in a leaf or if an apple has a bruise starting under the skin.

3. The Brain: The "Digital Time Machine" (NeRF)

This is where the AI comes in. They use a technology called NeRF (Neural Radiance Fields).

• The Analogy: Imagine you have 60 photos of a spinning apple taken from slightly different angles. A normal computer might try to stitch them together like a puzzle. But NeRF is like a digital time machine. It learns the "rules" of how light bounces off that specific apple. It builds a virtual 3D cloud of the apple in the computer's memory.
• The Twist: Usually, NeRF only knows about shape and color. This new method teaches the AI to also remember the 204 different chemical colors for every single point in that 3D cloud.

4. The Result: A "Digital Twin" You Can Inspect

The result is a 3D Hyperspectral Point Cloud.

• What is it? It's a digital twin of the fruit. You can spin it on your screen, zoom in, and look at any specific spot.
• Why is it cool? You can click on a tiny spot on the apple and ask the computer: "What is the chemical signature of this exact pixel?" The computer tells you, "This spot has high water content but is starting to bruise," even if you can't see the bruise with your naked eye yet.

Why Does This Matter?

• No More Moving Parts: Because the camera doesn't move, the system is cheaper, faster, and easier to build for factories.
• Better Food: Farmers and grocery stores can use this to automatically sort fruit. They can find the "bad apples" (literally) before they rot, saving money and reducing waste.
• Better Breeding: Scientists can breed better crops by seeing exactly how the inside of a plant changes as it grows, without having to cut the plant open.

In a Nutshell

The authors figured out how to take a stationary super-camera, spin the fruit in a glow-in-the-dark Teflon room, and use AI to build a 3D model that knows not just the shape of the fruit, but its chemical health at every single point. It's like giving a robot the ability to "see" the invisible biology of food.

Technical Summary

1. Problem Statement

High-throughput agricultural phenotyping and postharvest inspection require the simultaneous characterization of 3D geometric structure (morphology) and hyperspectral data (biochemical composition).

• Limitations of Current Methods: Traditional approaches combine hyperspectral imaging (HSI) with 3D reconstruction techniques like Structure-from-Motion (SfM) or LiDAR. These often rely on complex, expensive hardware setups (multi-camera arrays, moving gantries) that are difficult to scale, calibrate, and integrate into automated conveyor systems.
• Limitations of Standard NeRF: While Neural Radiance Fields (NeRF) offer efficient 3D reconstruction, standard implementations typically require moving cameras around a stationary object. This is logistically challenging in industrial agricultural settings where objects move on conveyors or turntables, and moving-camera rigs introduce synchronization and calibration issues.
• The Gap: There is a lack of scalable, stationary-camera frameworks capable of generating high-fidelity 3D hyperspectral point clouds (PCD) for automated agricultural workflows.

2. Methodology: HSI-SC-NeRF

The authors propose HSI-SC-NeRF, a framework that reverses the standard NeRF setup: the camera remains stationary while the object rotates. The pipeline consists of three main stages:

A. Experimental Setup & Data Acquisition

• Hardware: A stationary SPECIM IQ hyperspectral camera (204 bands, 400–1000 nm) is mounted on a tripod.
• Imaging Chamber: Objects are placed on a motorized turntable inside a custom-built Teflon (PTFE) chamber. The PTFE walls, combined with 12 tungsten halogen lamps, create diffuse, uniform illumination to minimize shadows and specular highlights.
• Acquisition Protocol: The object rotates 360°, and the camera captures one frame every 2 minutes (60 frames total per object).
• Pose Estimation: Since the camera is stationary, viewpoint variation is derived solely from object rotation. ArUco markers are placed on the turntable and a custom 3D-printed cylindrical container holding the object. COLMAP (SfM) is used to estimate camera poses relative to these markers, which are then transformed into the camera's frame of reference.

B. Data Preprocessing

• Spectral Calibration: A white reference (WR) tarp is used to correct for wavelength-dependent illumination bias and sensor response. A spatial mask is automatically generated to exclude edge artifacts, and the raw hyperspectral cubes are normalized and clipped to the [0, 1] reflectance range.
• Pose Transformation: The estimated poses (originally in the object/marker frame) are mathematically transformed to simulate a moving camera perspective, enabling standard NeRF training.

C. Multi-Channel NeRF Reconstruction

The core algorithm extends the Nerfacto pipeline (from NeRFStudio) to handle hyperspectral data:

• Model Architecture: The network predicts an $n$-dimensional spectral radiance vector ($c_\lambda$) and a scalar, wavelength-invariant transmittance ($\sigma$). This simplifies learning by decoupling geometry from spectral variation.
• Two-Stage Training Protocol:
  1. Pre-training: The model is trained on full-frame images (20k iterations) to establish a stable geometric initialization and coarse spectral alignment.
  2. Fine-tuning: The model is fine-tuned (20k iterations) using foreground masks to focus supervision only on the object pixels. Camera optimization is disabled during this stage to decouple radiometric refinement from pose updates.
• Loss Function: A composite loss function is used:
  • $L_{hsi}$ (Spectral Loss): Mean Squared Error (MSE) across all spectral bands.
  • $L_{ang}$ (Angular Loss): Penalizes discrepancies in spectral shape (cosine similarity) independent of magnitude.
  • Auxiliary Losses: Proposal loss, distortion regularization, orientation loss, and predicted-normal loss (standard Nerfacto terms) to ensure geometric stability.

3. Key Contributions

1. Stationary-Camera Pipeline: A novel workflow using ArUco-based pose estimation and simulated pose transformation to enable multi-view hyperspectral acquisition without moving the camera.
2. Custom Imaging Chamber: A Teflon-based chamber providing diffuse, uniform illumination, ensuring consistent spectral data acquisition essential for high-throughput agriculture.
3. Multi-Channel NeRF Formulation: An extended NeRF architecture with a composite spectral loss and a two-stage training protocol that jointly optimizes geometry and spectral fidelity across 204 bands.
4. Quantitative Validation: Comprehensive evaluation on three diverse agricultural samples (Apple, Pear, Maize) demonstrating high spatial accuracy and spectral fidelity.

4. Results

The framework was evaluated on an Apple, a Pear, and a Maize ear.

• Spatial Accuracy:
  • Using a ground-truth reference (reconstructed via a moving RGB camera), the stationary-camera NeRF achieved an F-score of 97.31% at a 2mm threshold.
  • The RMS point-to-point error after ICP registration was 0.001 m.
• Spectral Fidelity:
  • SAM (Spectral Angle Mapper): Values were consistently low (e.g., 0.029 rad for Maize), well below the 0.1 rad threshold for reliable classification.
  • Spectral RMSE: Low error rates across the 204 bands.
  • PSNR/SSIM: Values were competitive with standard NeRF benchmarks, though the authors emphasize SAM and RMSE as more critical for hyperspectral tasks.
• Loss Weight Ablation:
  • The study found that a hybrid loss is optimal.
  • Pre-training: Benefits from angular-dominant weighting to stabilize spectral shape.
  • Fine-tuning: Benefits from magnitude-dominant weighting (specifically $\lambda_{ang}=0.25, \lambda_{hsi}=0.75$) to correct radiometric intensity errors once geometry is fixed.
• Application-Specific Visualization:
  • The system successfully highlighted biologically relevant features (e.g., apple bruises, maize kernel dryness, pear disease) by selecting specific wavelength triplets (e.g., 801, 708, 551 nm for apple bruising).
  • Band selection influenced mesh-derived traits (surface area/volume) in apples but remained stable for maize and pear, demonstrating the sensitivity of geometric extraction to spectral contrast.

5. Significance and Impact

• Scalability: By eliminating complex moving-camera rigs, HSI-SC-NeRF makes high-throughput 3D hyperspectral imaging feasible for automated conveyor systems in postharvest facilities and breeding programs.
• Non-Destructive Quality Control: The ability to simultaneously reconstruct 3D geometry and biochemical signatures (water content, chlorophyll, defects) allows for early detection of stress, disease, and bruising without damaging the produce.
• Cost-Effectiveness: The use of a single stationary camera and a simple turntable reduces hardware costs and operational complexity compared to multi-camera arrays or LiDAR setups.
• Future Directions: While the method assumes rigid-body motion and requires careful calibration, it provides a robust foundation for integrating physically meaningful spectral data into automated agricultural workflows, potentially reducing food waste and accelerating breeding cycles.