Long-Term Multi-Session 3D Reconstruction Under Substantial Appearance Change

This paper presents a joint Structure-from-Motion framework that integrates handcrafted and learned visual features to robustly establish cross-session correspondences, enabling the creation of coherent long-term 3D models from imagery captured years apart despite substantial appearance and structural changes where existing methods fail.

The Gist

Imagine you are trying to build a single, perfect 3D model of a coral reef, but you are taking photos of it over three different years.

In 2016, the reef looks one way. In 2017, a storm hits, and the coral shifts. By 2018, the colors have changed, and some structures are gone. Your goal is to stitch all these photos together into one coherent map so scientists can see exactly how the reef changed over time.

This is the problem the paper solves. Here is how they did it, explained simply.

The Problem: The "Puzzle" That Doesn't Fit

Think of standard 3D mapping software (like the kind used to make Google Earth views) as a puzzle solver.

• The Old Way: The old software tries to solve the 2016 puzzle, then the 2017 puzzle, and the 2018 puzzle separately. Once they are done, it tries to glue the three finished puzzles together.
• The Failure: Because the reef changed so much (colors shifted, rocks moved), the "edges" of the 2016 puzzle don't match the 2017 puzzle. The software gets confused, thinks they are different places, and the final map ends up broken, misaligned, or missing huge chunks. It's like trying to glue a picture of a puppy to a picture of a grown dog and expecting them to fit perfectly.

The Solution: The "Master Architect" Approach

The authors say, "Don't build the puzzles separately and try to glue them later. Build the whole thing at once, from the start."

They created a new method that acts like a Master Architect who looks at all the photos from all three years simultaneously. Instead of waiting until the end to see if they fit, the Architect forces the pieces to align while building the model.

Here are the three secret ingredients they used:

1. The "Hybrid Detective" (Handcrafted + Learned Features)

To find matching points between photos, the software needs to recognize things.

• The Old Detective: Uses simple, rigid rules (like "look for a corner"). This works great for photos taken minutes apart, but fails when the reef looks totally different years later.
• The New Detective: Uses a mix of old-school rules and a super-smart AI brain (Deep Learning). The AI is good at recognizing that "this weird shape in 2016 is actually the same rock as this blurry blob in 2018," even if they look nothing alike.

2. The "Smart Filter" (Visual Place Recognition)

You might think, "Okay, let's just have the AI compare every photo from 2016 with every photo from 2018."

• The Problem: That would take forever. It's like trying to find a specific needle in a haystack by checking every single straw in the world.
• The Fix: They use a "Smart Filter" (Visual Place Recognition) first. This filter quickly scans the photos and says, "Hey, Photo A from 2016 and Photo B from 2018 look like they were taken in the same neighborhood."
• The Result: The AI only does its heavy lifting on those specific pairs. It's like only asking the detective to investigate the suspects who were actually at the scene, rather than interviewing the whole city. This saves massive amounts of time and computing power.

3. The "Joint Optimization" (Building Together)

Instead of building three separate models and trying to jam them together, the software builds one giant model using all the photos at once.

• It forces the 2016 camera positions and the 2018 camera positions to agree on where the rocks are, even if the rocks look different.
• If the software sees a mismatch, it adjusts the whole map slightly to make everything fit, rather than giving up.

The Result: A Time-Traveling Map

When they tested this on real coral reefs in Japan (which had been battered by typhoons and changed over three years):

• Old methods: Failed completely. The maps were broken, or the software couldn't find enough matches to build anything.
• Their method: Successfully created a single, smooth 3D map where a photo from 2016 and a photo from 2018 lined up perfectly, pixel-by-pixel.

Why This Matters

Imagine you are a doctor trying to track a patient's healing over years. If your X-ray machine can't align the 2016 scan with the 2024 scan because the patient's body changed shape, you can't see the progress.

This paper gives scientists a way to "align the X-rays" of the ocean. It allows them to:

• See exactly how much coral has died or grown.
• Measure damage from storms accurately.
• Plan how to save reefs based on real, aligned data.

In short: They stopped trying to force mismatched puzzle pieces together at the end. Instead, they built the whole puzzle at once using a smart AI that knows how to recognize things even when they've changed, all while using a filter to keep the process fast enough to actually run on a computer.

Technical Summary

1. Problem Statement

The paper addresses the challenge of reconstructing a single, coherent 3D model from visual surveys of natural environments (specifically coral reefs) captured months or years apart.

• The Core Issue: Existing Structure-from-Motion (SfM) pipelines implicitly assume near-simultaneous image capture with limited appearance variation. When applied to long-term monitoring, these pipelines fail because:
  • Substantial Change: Environments like coral reefs undergo significant structural and visual changes due to seasons, storms (e.g., typhoons), and biological growth.
  • Failure of Post-Hoc Alignment: The standard strategy of reconstructing each session independently and then aligning the resulting point clouds (post-hoc) fails. Without persistent geometric landmarks, the alignment is brittle, often resulting in fragmented models or misaligned geometries that lack pixel-level accuracy.
  • Limitations of Joint Reconstruction: Standard joint SfM approaches also fail because they rely on feature matching techniques that cannot robustly establish correspondences across large temporal gaps and severe appearance changes.

2. Methodology

The authors propose a Hybrid Joint SfM Pipeline that enforces cross-session correspondences directly within the optimization process, rather than attempting to align separate models later.

Key Components:

• Joint Optimization Framework:
  • Instead of separate reconstructions, images from all survey visits (e.g., 2016, 2017, 2018) are fed simultaneously into a single SfM pipeline (based on COLMAP).
  • Camera poses, intrinsics, and 3D structure are jointly estimated, ensuring all data exists in a shared coordinate frame.
• Hybrid Feature Matching Strategy:
  • Intra-Session (Same Visit): Uses fast, exhaustive matching of handcrafted SIFT features. Since appearance changes are minimal within a single dive, this ensures dense and reliable local reconstruction.
  • Cross-Session (Different Visits): Uses learned feature matching (LightGlue) to handle substantial appearance changes.
• Visual Place Recognition (VPR) for Scalability:
  • Applying learned matching exhaustively across all image pairs is computationally prohibitive.
  • The authors use MegaLoc (a VPR method) to generate a distance matrix and identify candidate cross-session image pairs that likely view the same physical location.
  • LightGlue is applied only to these candidates. This drastically reduces computational cost while focusing robust matching where it is most needed.
• Roll Robustness:
  • Given the downward-facing camera on the AUV, images can have arbitrary rotations. The system explicitly evaluates discrete rotations (0°, 90°, 180°, 270°) for both VPR and feature matching to ensure correct alignment.

3. Key Contributions

1. Conceptual Insight: The paper identifies that post-hoc alignment of independently reconstructed sessions is insufficient for long-term monitoring under substantial change. It argues that cross-session constraints must be enforced during the SfM optimization.
2. Methodological Innovation: Introduction of a hybrid pipeline combining handcrafted (SIFT) and learned (LightGlue) features, gated by Visual Place Recognition. This enables the creation of a single coherent 3D model from imagery captured years apart, a task where standard pipelines fail.
3. Empirical Validation: Evaluation on a real-world, multi-year coral reef dataset (Sesoko Island, 2016–2018) exhibiting severe structural disruption (post-typhoon). The method demonstrates consistent joint reconstruction where existing state-of-the-art methods produce fragmented or misaligned results.

4. Experimental Results

The method was evaluated on a dataset of underwater surveys spanning three years, including a period before and after Typhoon Trami.

• Quantitative Performance (Reprojection Error):
  • Baseline (Post-hoc Alignment): Standard COLMAP + ICP alignment resulted in a median reprojection error of 165.26 pixels. COLMAP + BUFFER-X improved this to 77.11 pixels, but errors remained high.
  • Proposed Method: Achieved a median reprojection error of 3.65 pixels.
  • Improvement: This represents a 97.8% improvement over post-hoc ICP and a 95.3% improvement over BUFFER-X.
• Qualitative Performance:
  • COLMAP (Joint): Failed to incorporate all sessions, resulting in incomplete point clouds due to insufficient cross-session matches.
  • MapAnything (End-to-End): Produced fragmented geometry with visible misalignments between overlapping regions.
  • Proposed Method: Successfully integrated imagery from all three years into a single, coherent 3D point cloud with pixel-level alignment accuracy.
• Computational Efficiency:
  • Exhaustive learned matching for 2,500 images took ~57 hours.
  • Using VPR to filter candidates reduced this to 2.5 hours (a 95.5% reduction in time).
  • Crucially, the VPR-filtered approach yielded better accuracy (3.65 px error) than exhaustive matching (6.33 px error), likely by reducing the influence of noisy, spurious matches.

5. Significance and Impact

• Enabling Long-Term Monitoring: This approach solves a critical bottleneck in autonomous environmental monitoring. It allows researchers to track structural changes (e.g., coral bleaching, storm damage, restoration progress) over years with high geometric fidelity.
• Robustness in Unstructured Environments: The method proves effective in "unstructured" natural environments where traditional geometric landmarks are absent or changing, challenging the assumption that post-hoc alignment is a viable strategy for long-term SLAM.
• Scalability: By combining VPR with learned matching, the system remains computationally tractable for large datasets, making it suitable for deployment in real-world, long-duration autonomous missions.
• Application: The technology directly supports marine conservation efforts, such as the designation of Marine Protected Areas (MPAs) and active reef restoration planning, by providing a reliable, unified 3D record of ecosystem health over time.