💻 computer science

Practical High-Fidelity Novel-View Synthesis of Mounted Lepidoptera

This paper presents an end-to-end pipeline that combines handheld focus stacking, a non-contact mirror system, and a mirror-aware 3D Gaussian Splatting extension to generate photo-realistic 3D models of mounted butterflies, overcoming challenges related to their fragility, microscopic details, and inaccessible ventral surfaces.

Original authors: Kristof Overdulve, Lode Jorissen, Nick Michiels

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kristof Overdulve, Lode Jorissen, Nick Michiels

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a beautiful, fragile butterfly pinned in a museum case. It's a masterpiece of nature, covered in tiny, iridescent scales and delicate veins. Now, imagine you want to create a perfect 3D digital twin of it so people can spin it around and look at it from every angle on a computer.

This is incredibly difficult for three main reasons:

It's tiny: To see the tiny details, you need a super-magnifying lens (a macro lens). But these lenses are like looking through a very narrow tunnel; only a tiny slice of the butterfly is in focus at once, while the rest is blurry.
It's pinned down: The butterfly is stuck to a board by a pin. You can't lift it up to take a picture of its belly (the underside of the wings) because it's too fragile to touch.
It's one-sided: Since you can't lift it, a normal camera can only see the top. But the top and bottom of a butterfly often look completely different.

This paper presents a clever "pipeline" (a step-by-step recipe) to solve these problems and create a photo-realistic 3D model that you can look at from any angle, including the underside. Here is how they did it, using simple analogies:

1. The "Magic Mirror" Trick (Solving the "Can't Touch It" Problem)

Since they couldn't lift the butterfly to see its belly, they used a first-surface mirror (like the kind dentists use, where the reflective coating is on the very front, not behind glass).

The Analogy: Imagine the butterfly is sitting on a table. Instead of picking it up, the researchers placed two mirrors on the table next to it, angled just right.
The Result: When the camera looks at the butterfly, it sees the real top and the reflection of the belly in the mirror. It's like having a second camera underneath the table without ever touching the insect.

2. The "Handheld Focus Stack" (Solving the "Blurry Lens" Problem)

Because the macro lens only keeps a tiny slice in focus, a single photo would be mostly blurry. Usually, scientists use heavy tripods and motorized rails to take 20 photos, shifting the focus slightly each time, and then stitch them together. But setting up heavy equipment is slow and risky for fragile specimens.

The Analogy: Think of taking a stack of photos with your phone while holding it slightly shaky. The researchers did this too! They took a burst of 20 photos by hand, moving the focus slightly with each shot.
The Fix: Their software is smart enough to "stabilize" the shaky hand movements (like a digital gimbal) and then blend the 20 photos together. It picks the sharpest part of every single photo and combines them into one perfectly sharp, all-in-focus image.

3. The "Mirror-Aware" 3D Artist (Solving the "Confused Computer" Problem)

This is the most technical part, but here's the simple version. They used a modern 3D reconstruction technology called 3D Gaussian Splatting. Imagine this technology as a digital artist who builds a 3D model out of millions of tiny, fuzzy, colored clouds (Gaussians).

The Problem: If you just feed the camera photos (which include the mirror reflection) to a standard 3D artist, the computer gets confused. It thinks the reflection in the mirror is a second, real butterfly floating behind the glass. It ends up building two butterflies: one real, one fake ghost.
The Solution: The researchers taught the computer to recognize the mirror.
1. Detecting the Mirror: The computer looks at the 3D "clouds" it built and notices they are perfectly symmetrical (like a reflection). It calculates exactly where the mirror plane is.
2. The "Fold": Instead of letting the computer build a second ghost butterfly, the system takes the real butterfly, folds it over the mirror line, and creates a perfect reflection mathematically.
3. The Result: The computer now has one single, consistent butterfly. When you look at the "real" side, you see the top. When you look at the "mirror" side, you see the bottom. It's all one object, not two.

The Final Outcome

By combining these three steps, the team created a system that:

Doesn't require expensive, heavy equipment (you can do it handheld).
Doesn't require touching the fragile specimen (using mirrors).
Doesn't need a human to manually cut out the butterfly from the background (the math handles the mirror automatically).

The result is a high-fidelity, photo-realistic 3D model of a pinned butterfly that you can spin around and inspect from the top, bottom, and every angle in between, preserving the delicate details that usually get lost in digitization. The authors plan to use these models in a virtual museum experience.

Technical Summary: Practical High-Fidelity Novel-View Synthesis of Mounted Lepidoptera

Problem Statement

Digitizing mounted Lepidoptera (butterflies and moths) for natural history collections presents a unique set of challenges that existing 3D scanning workflows fail to address. These specimens are small, fragile, and covered in microscopic structures (hairs, scales, veins) that require macro-lens photography. However, macro lenses possess an extremely limited Depth of Field (DoF), rendering most of the specimen out of focus in a single shot. Furthermore, standard camera bodies cannot be maneuvered beneath a pinned specimen to capture the ventral (underside) surface without physically removing the specimen, which risks irreparable damage to the delicate wing-to-thorax attachment. Finally, synthetic mirroring of the dorsal surface is insufficient because dorsal and ventral patterns often differ entirely. The goal is to create a photo-realistic 3D model viewable from any direction without physically touching or moving the fragile specimen.

Methodology

The authors propose an end-to-end pipeline combining three core components to overcome these physical and computational constraints:

1. Non-Contact Mirror System

To capture the ventral surface without touching the specimen, the system utilizes two first-surface mirrors placed symmetrically around the specimen's pin on a foam substrate. Unlike second-surface mirrors, first-surface mirrors place the reflective coating on the front face, eliminating ghost reflections and color fringing. This setup exposes the ventral surface to the camera via reflection while keeping the specimen stationary and undisturbed.

2. Handheld Focus Stacking with Automatic Registration

To address the limited DoF of macro lenses, the pipeline employs a handheld focus stacking approach rather than a tripod-mounted or motorized rail system.

Capture: At each viewpoint, the photographer captures a burst of $K=20$ frames at continuous drive speed, shifting focus from the nearest to the farthest part of the specimen.
Registration: Because the camera is handheld, consecutive frames exhibit jitter. The system aligns these frames using the Enhanced Correlation Coefficient (ECC) algorithm. To optimize contrast for alignment, the RGB frames are projected onto a grayscale axis derived from the first eigenvector of the pixel covariance matrix (PCA).
Fusion: Aligned frames are merged into a single all-in-focus image using a multi-resolution pyramid fusion method (inspired by Laplacian pyramid blending), selecting the sharpest details from each source frame at every scale.

3. Segmentation-Free Mirror-Aware 3D Gaussian Splatting (3DGS)

The core reconstruction engine is an extension of 3D Gaussian Splatting designed to handle mirror reflections without requiring per-pixel segmentation masks.

Mirror Plane Detection: Instead of manual segmentation, the system exploits the bilateral symmetry of the dense point cloud generated by Multi-View Stereo (MVS). It creates an artificial "mirror clone" of the point cloud, registers it back to the original using feature matching (FPFH) and RANSAC followed by Point-to-Plane ICP, and extracts the mirror plane parameters (normal and offset) from the resulting rigid transformation.
Gaussian Reflection: Once the mirror plane is identified, the system reflects all 3D Gaussians across this plane. The reflection involves:
- Reflecting the mean position ( $\mu$ ).
- Reflecting the orientation (quaternion) by applying a Householder reflection matrix and correcting the determinant to ensure a valid rotation.
- Copying scales, opacities, and spherical harmonic coefficients unchanged.
Training: The combined set of original and reflected Gaussians is rasterized simultaneously. The system is trained on both direct views of the specimen and views containing the mirror reflection, learning a single, physically consistent 3D representation.

Key Contributions

Non-Contact Mirror Imaging: A practical method for imaging the ventral surface of pinned Lepidoptera using first-surface mirrors, avoiding physical manipulation of fragile specimens.
Handheld Focus Stacking Pipeline: A robust workflow that compensates for hand jitter via automatic image registration, enabling all-in-focus macro capture without specialized motorized hardware or tripods.
Segmentation-Free Mirror-Aware 3DGS: A novel extension of 3D Gaussian Splatting that automatically detects the mirror plane from dense geometry and reflects Gaussians to reconstruct a single, consistent object. This eliminates the need for manual segmentation masks, which are difficult to generate for fine structures like hairs and antennae.

Results and Evaluation

The pipeline was validated on four diverse butterfly specimens with varying sizes, colors, and mounting types.

Macro Defocus Handling: Quantitative evaluation (PSNR, SSIM, LPIPS) demonstrated that the proposed "acquire-then-fuse" focus stacking strategy significantly outperforms depth-of-field-aware 3DGS methods (e.g., DOF-GS) on macro data. The DoF-aware methods failed to recover high-frequency details (scales and veins), resulting in blurry reconstructions, whereas focus stacking produced sharp, all-in-focus renders.
Mirror-Aware Reconstruction: The authors compared their mirror-aware 3DGS against a baseline 3DGS (and a post-hoc folding approach that performed poorly due to background artifacts).
- On held-out dorsal views (the training distribution), the mirror-aware method achieved metrics comparable to the unconstrained baseline. The authors note this is expected, as the baseline can fit the dorsal view and the mirror reflection as two independent objects.
- The decisive advantage appears in ventral views (outside the training distribution). The baseline disintegrates into two disconnected objects, failing to render the underside correctly. In contrast, the proposed method maintains a single, coherent butterfly that renders correctly from any viewpoint, including the ventral side.

Significance and Claims

The paper claims to present a practical, reproducible, and end-to-end solution for the 3D digitization of mounted Lepidoptera, a task previously hindered by the fragility of specimens and the optical limitations of macro photography. By combining established techniques (focus stacking, MVS) with a novel, segmentation-free geometric approach to mirror handling in 3DGS, the system produces full-surround, photo-realistic 3D models. The authors state that these models are suitable for deployment in virtual museum experiences and open-access scientific repositories, addressing a critical gap in the digitization of natural history collections. The work emphasizes that the method is specifically tailored to the "uniquely challenging middle ground" of pinned insects, where standard workflows for flat sheets or robust 3D objects are insufficient.