DL-VINS-Factory: A Modular Framework for Learned Visual Front-Ends in Visual-Inertial SLAM

Technical Summary: DL-VINS-Factory

Problem Statement
While deep learning-based feature extractors have demonstrated superior performance in visual matching compared to handcrafted features (e.g., ORB, BRISK), their practical integration into tightly coupled Visual-Inertial Simultaneous Localization and Mapping (VI-SLAM) systems remains fragmented. Existing studies often evaluate learned extractors under disparate back-end optimizers, loop-closure strategies, and hardware platforms, making it difficult to isolate the specific impact of the visual front-end. Furthermore, traditional loop-closure systems rely on descriptor-specific Bag-of-Words (BoW) vocabularies, creating a hard coupling that necessitates rebuilding the entire vocabulary whenever the underlying feature representation changes. This fragmentation hinders fair benchmarking and the systematic adoption of emerging learned features in real-time embedded robotics.

Methodology
The authors present DL-VINS-Factory, a modular VI-SLAM framework designed to decouple learned visual front-ends from the estimation back-end and loop-closure pipeline. The system is built upon a re-implementation of VINS-Fusion, featuring a unified sliding-window Ceres back-end and 4-DoF Pose Graph Optimization (PGO).

Key methodological components include:

• Modular Front-Ends: The framework supports interchangeable learned feature extractors: ALIKED, RaCo, SuperPoint, and XFeat. These can be paired with either Lucas-Kanade (LK) optical-flow tracking or LightGlue (LG) descriptor matching.
• Descriptor Agnosticism: To address the coupling issue in loop closure, the system integrates AnyLoc, which uses DINOv2 patch embeddings aggregated via a universal Vector of Locally Aggregated Descriptors (VLAD) codebook. This allows for image-level retrieval independent of the specific local feature descriptor used for tracking. Geometric verification reuses the active local descriptors (via brute-force matching, Fundamental Matrix RANSAC, and PnP RANSAC).
• Implementation Details:
  • RaCo Integration: Since RaCo detects keypoints but does not emit descriptors, the framework decomposes ALIKED to supply descriptors for RaCo keypoints, creating a hybrid RaCo+ALIKED descriptor stream.
  • Optimization: Custom CUDA kernels replace dynamic stages (e.g., ALIKED's DKD) to enable static-shape TensorRT engines. A fused pre-processing kernel handles padding and format conversion to minimize host-device transfer overhead.
  • Hardware: The system is evaluated on an NVIDIA RTX 3080 Ti workstation and an NVIDIA Jetson AGX Orin, utilizing TensorRT for acceleration.

Key Contributions

1. Unified Framework: Development of a modular VI-SLAM system enabling controlled comparison of various feature extractor and matcher combinations (ALIKED, RaCo, SuperPoint, XFeat with LK or LightGlue) within a single tightly coupled architecture.
2. Front-End Agnostic Loop Closure: Integration of a loop-closure module using AnyLoc (DINOv2-VLAD) that eliminates the need for per-descriptor BoW vocabulary construction, facilitating easier switching of front-ends.
3. Systematic Evaluation: Extensive benchmarking across four diverse datasets (EuRoC, NTU-VIRAL, Botanic Garden, SubT-MRS) covering indoor, outdoor, aggressive-motion, and visually degraded conditions on both workstation and embedded hardware.

Results
The evaluation reveals that no single front-end configuration is universally superior; performance is heavily dependent on scene structure, motion profiles, and sensor configuration.

• Accuracy Trends:
  • EuRoC (Structured Indoor): Classical baselines (GFTT+LK) remain highly competitive. Learned front-ends like ALIKED+LightGlue show modest improvements (5–7% reduction in ATE) over baselines in monocular and stereo modes.
  • NTU-VIRAL (Aggressive Motion): Learned matching (LightGlue) significantly outperforms optical flow due to large inter-frame viewpoint changes. ALIKED+LightGlue stereo reduced loop-closed ATE by 12% compared to the baseline.
  • Botanic Garden (Unstructured Vegetation): Optical flow tracking (LK) generally outperforms LightGlue matching, likely due to low texture and weak stereo correspondences. However, learned keypoints still improve over handcrafted baselines (e.g., SuperPoint+LK reduced grayscale ATE by 29%).
  • SubT-MRS (Degraded Conditions): In smoke and darkness, learned descriptors paired with optical flow remained robust, while LightGlue variants often failed due to reliance on high-confidence descriptor correspondences which vanish under severe occlusion.
• Loop Closure: AnyLoc (DINOv2-VLAD) detected 2–7× more valid loop candidates than the traditional BRIEF+DBoW2 baseline. However, the cosine similarity gate alone had low precision, requiring geometric verification to filter false positives.
• Real-Time Performance: With TensorRT acceleration on Jetson AGX Orin, all valid configurations ran in real-time (29–47 FPS monocular, 18–33 FPS stereo).
• Ablation Studies:
  • Keypoint Count: A cap of 256 keypoints was found to be optimal for ALIKED and RaCo, balancing CUDA kernel efficiency and feature sufficiency.
  • Covariance Weighting: For RaCo, utilizing the covariance head for anisotropic weighting improved stereo accuracy significantly (up to 80% ATE reduction in specific failure cases) but had neutral or negative effects in monocular mode.

Significance and Claims
The paper claims that DL-VINS-Factory successfully isolates the impact of visual front-ends, demonstrating that learned features are viable for real-time embedded VI-SLAM but are not universally superior to classical tracking. The choice of front-end must be conditional on the deployment environment:

• Learned matching (LightGlue) is advantageous for large viewpoint changes and aggressive motion.
• Optical flow (LK) remains preferable for low-parallax trajectories and visually degraded or textureless environments.
• Stereo vision improves accuracy only when metric depth is reliably observable; in low-texture vegetation, it can degrade performance due to weak constraints.

The work emphasizes that front-end representation alone does not dictate system performance; scene geometry, motion excitation, and back-end conditioning are equally critical. The framework provides a standardized path for integrating and evaluating new learned features without the engineering overhead of rebuilding entire SLAM pipelines.