LST-SLAM: A Stereo Thermal SLAM System for Kilometer-Scale Dynamic Environments

The paper proposes LST-SLAM, a novel stereo thermal SLAM system that integrates self-supervised feature learning, dual-level motion tracking, and semantic-geometric constraints to achieve robust, accurate localization and mapping in kilometer-scale dynamic outdoor environments, significantly outperforming existing state-of-the-art methods.

The Gist

Imagine you are trying to navigate a car through a city, but you have to do it in pitch darkness, through heavy fog, and while other cars and people are constantly moving around you.

If you were using a normal camera (like the one on your phone), you would be blind. The low light would make everything gray and blurry, and the moving people would confuse your brain, making you think the road is shifting when it's not.

This is the exact problem robots face when they try to use thermal cameras (which see heat instead of light). Thermal cameras are great for seeing in the dark or fog, but they have their own problems: the images look like "static" on an old TV (low contrast), and the heat signatures of moving objects (like cars) mess up the robot's ability to know where it is.

The paper introduces LST-SLAM, a new "brain" for robots that solves these problems. Here is how it works, using simple analogies:

1. The Problem: "The Static TV"

Think of a thermal image as a black-and-white TV screen with a lot of static.

• Old methods tried to find "corners" and "edges" in this static, just like they do with normal photos. But because thermal images are so blurry and noisy, the robot kept losing its place, like trying to find a specific grain of sand on a beach during a storm.
• Moving objects (like a bus driving by) act like ghosts. If the robot tries to track the bus, it thinks the whole world is moving, which throws off its map.

2. The Solution: LST-SLAM's Superpowers

The authors built a system with four main "superpowers" to fix this:

A. The "Heat-Smart" Eye (Self-Supervised Learning)

• The Analogy: Imagine teaching a child to recognize shapes. Usually, you show them thousands of color photos. But here, the robot has to learn to recognize shapes in "heat photos" where there are no colors.
• How it works: The system takes a pre-trained AI (SuperPoint) that is already an expert at seeing shapes in normal photos. It then "fine-tunes" this AI specifically for thermal images. It teaches the robot to ignore the "static" and focus on the unique heat patterns of buildings and roads. It's like giving the robot a pair of specialized glasses that turn fuzzy heat blobs into sharp, trackable landmarks.

B. The "Double-Check" Tracker (Stereo Dual-Level Tracking)

• The Analogy: Imagine you are walking through a crowd.
  • Level 1 (Photometric): You look at the shadows and brightness of people to guess where they are.
  • Level 2 (Descriptor): You also look at their clothing patterns and faces to confirm.
• How it works: The robot doesn't just rely on one way to track movement. It uses two methods at once: one that checks the brightness of the heat image, and another that checks the "fingerprint" of the features. If one method gets confused by a moving car, the other keeps the robot on track.

C. The "Ghost Buster" (Dynamic Feature Filtering)

• The Analogy: You are trying to navigate a city, but a parade is happening. If you try to follow the marching band, you'll think the city is spinning. You need to ignore the parade and only look at the buildings.
• How it works: The system uses a smart detector (YOLO) to spot moving things like cars and people. It puts a "Do Not Track" sticker on them. It then checks if the remaining "static" objects (buildings, trees) are moving in a way that makes geometric sense. If a "building" seems to be sliding across the screen, the system realizes it's actually a moving object masquerading as a building and throws it out.

D. The "Memory Book" (Loop Closure)

• The Analogy: Imagine you are walking in a giant, foggy forest for 10 miles. You start to drift off course. Suddenly, you see a tree you recognize from 2 miles ago. You realize, "Ah! I've been walking in a circle!" and you correct your map.
• How it works: As the robot travels kilometers, it builds a "word book" (Bag-of-Words) of the unique heat patterns it sees. When it sees a familiar pattern again, it knows it has returned to a previous spot. It then uses this "aha!" moment to fix all the small errors that have piled up over the last 10 miles, snapping the map back into perfect alignment.

3. The Results: Why It Matters

The researchers tested this system on real roads, driving for kilometers in day, night, and bad weather.

• The Competition: They compared it to other top-tier robot navigation systems (like AirSLAM and DROID-SLAM).
• The Winner: LST-SLAM was significantly more accurate. It made 75% fewer mistakes than the next best thermal system.
• The Takeaway: While other systems would get lost or confused in the dark or fog, LST-SLAM kept its cool, built a perfect map, and knew exactly where it was.

Summary

LST-SLAM is like giving a robot a super-powered night vision, a smart filter to ignore moving traffic, and a perfect memory to correct its own mistakes. It allows robots to drive themselves safely in the dark, in the fog, and in busy cities where normal cameras would fail completely.

Technical Summary

1. Problem Statement

Visual Simultaneous Localization and Mapping (SLAM) using RGB cameras often fails in challenging conditions such as low light, fog, smoke, or sudden illumination changes. Thermal cameras offer a robust alternative by capturing infrared radiation, but Thermal SLAM faces unique and significant hurdles:

• Feature Instability: Thermal images suffer from low contrast, weak textures, and high noise, making traditional feature extraction (designed for RGB) unreliable.
• Inter-frame Inconsistency: Non-uniformity correction (NUC) in thermal cameras introduces temporal inconsistencies, hindering long-term tracking.
• Dynamic Environments: Large-scale outdoor scenes contain dynamic objects (vehicles, pedestrians) that cause visual aliasing and introduce outliers during pose optimization, degrading accuracy.
• Lack of Large-Scale Solutions: Existing thermal SLAM systems often rely on multi-sensor fusion (LiDAR, IMU, RGB) or are limited to small-scale/indoor environments. There is a lack of robust, thermal-only systems capable of operating at kilometer-scale in dynamic outdoor settings.

2. Methodology

The authors propose LST-SLAM, a stereo thermal SLAM system designed to address these challenges through four core components:

A. Thermal Image Preprocessing

Raw 16-bit thermal images are normalized to 8-bit using percentile stretching (1%–99%) with an Exponential Moving Average (EMA) to smooth intensity bounds over time, reducing flickering. Contrast-Limited Adaptive Histogram Equalization (CLAHE) is applied to enhance local contrast while suppressing noise.

B. Self-Supervised Thermal Feature Learning (STP Network)

To overcome the scarcity of thermal training data and the poor performance of RGB-trained models (like SuperPoint) on thermal data:

• Architecture: Based on the SuperPoint architecture (shared encoder, detection head, descriptor head).
• Transfer Learning: The network is initialized with weights from SuperPoint to transfer geometric priors from the RGB domain.
• Training Strategy: The detection head is frozen to preserve keypoint localization. Only the descriptor head is trained using a Thermal Homography Training Loss. This self-supervised approach uses homography-transformed image pairs to learn robust descriptors without requiring pseudo-labels.

C. Dynamic Feature Filtering

To mitigate the impact of moving objects in traffic scenes:

• Semantic-Geometric Hybrid Constraint: A YOLOv8-Seg network (pre-trained on RGB but effective on thermal due to structural cues) identifies dynamic regions (cars, buses, etc.).
• Geometric Verification: Keypoints within dynamic masks are retained only if they satisfy strict epipolar geometric consistency (distance to epipolar line < 0.5 pixels) via RANSAC. Keypoints outside dynamic masks rely on standard optical flow tracking.

D. Stereo Dual-Level Motion Tracking

The system employs a two-stage tracking strategy to balance robustness and efficiency:

1. Low-Level Photometric Tracking: Estimates coarse relative pose by minimizing patch-based photometric error (similar to semi-direct VO).
2. High-Level Descriptor Tracking: Uses binary descriptors (derived from the learned features via sign testing) for efficient Hamming distance matching. This is followed by PnP-LM refinement to optimize the camera pose.

E. Loop Closure and Global Optimization

• Incremental Online Bag-of-Words (iBoW): Instead of offline training, the visual dictionary is built incrementally online. Deep descriptors are binarized to enable efficient Hamming distance search.
• Loop Detection: A two-stage process involving appearance-based retrieval and geometric verification (fundamental matrix estimation).
• Global Optimization: Validated loops introduce constraints to the pose graph, followed by Sim(3) global bundle adjustment to correct accumulated drift and scale.

3. Key Contributions

1. LST-SLAM System: A novel, fully thermal stereo SLAM system tailored for kilometer-scale, dynamic, and illumination-challenging outdoor environments.
2. STP Network: A self-supervised thermal point network that adapts RGB geometric knowledge to the thermal domain using a specialized homography loss, significantly improving feature stability.
3. Hybrid Dynamic Filtering: A strategy combining semantic segmentation (YOLOv8) and geometric constraints to robustly filter dynamic outliers without requiring retraining on thermal data.
4. Incremental Thermal BoW: An online loop closure mechanism that adapts to varying thermal feature distributions without offline training.

4. Experimental Results

The system was evaluated on the MS2 dataset (kilometer-scale stereo-thermal sequences) and compared against state-of-the-art methods including AirSLAM, DROID-SLAM, VINS-Fusion, and ORB-SLAM3.

• Feature Extraction: The STP network achieved 35.8% to 48.4% more inlier matches compared to SuperPoint, SIFT, and ORB, demonstrating superior robustness in thermal domains.
• Localization Accuracy:
  • LST-SLAM achieved the lowest Absolute Trajectory Error (ATE) across all sequences (Morning, Daytime, Nighttime).
  • Comparison: It reduced localization error by 75.8% compared to AirSLAM and 66.8% compared to DROID-SLAM.
  • Robustness: Unlike other systems that failed to initialize or track in specific thermal conditions (e.g., SVO failing in low light), LST-SLAM maintained 100% trajectory coverage (CR metric) across all test sequences.
• Ablation Study: Removing any single module (STP, Loop Closure, Segmentation, or Dual-Level Tracking) resulted in significant performance degradation, confirming the necessity of the full pipeline.

5. Significance

This work represents a major step forward in thermal-only autonomous navigation. By solving the fundamental issues of feature instability and dynamic object interference in thermal imagery, LST-SLAM enables reliable robot perception in:

• Complete darkness and extreme weather (fog/smoke).
• Large-scale outdoor environments (kilometers long) without reliance on expensive multi-sensor fusion (LiDAR/IMU).
• Dynamic traffic scenarios, proving that thermal SLAM can be as robust as RGB-based systems when specifically engineered for the thermal modality.

The paper establishes a new benchmark for thermal SLAM, demonstrating that self-supervised learning and geometric constraints can overcome the inherent limitations of thermal imaging for long-term autonomous operation.