RoEL: Robust Event-based 3D Line Reconstruction

This paper proposes RoEL, a robust method for 3D line reconstruction and pose refinement using event cameras that leverages multi-temporal line tracking and specialized geometric cost functions to overcome data sparsity, noise, and domain discrepancies, thereby enabling effective multimodal mapping and localization.

The Gist

Imagine you are trying to draw a map of a room, but instead of using a standard camera that takes pictures like a human eye, you are using a special "event camera."

The Problem: The Flickering Firefly
Standard cameras take a photo every 1/30th of a second, capturing a full picture of the world. Event cameras are different. They don't take pictures; they act like a swarm of hyper-sensitive fireflies. They only "flash" when they see something change—like a door opening, a shadow moving, or a light flickering.

This is great for speed and battery life, but it creates a messy problem:

1. It's sparse: You don't get a full picture; you just get a few scattered flashes.
2. It's noisy: In the dark or when moving fast, these fireflies go crazy, flashing randomly (noise) or missing things entirely.
3. It's blurry: If you try to stack these flashes to make a "picture," the edges get smeared out, like trying to draw a straight line with a shaky hand.

Previous methods tried to force these messy flashes into 3D models, but the results were often jagged, full of errors, or just plain wrong.

The Solution: RoEL (Robust Event-based Line Reconstruction)
The authors of this paper, RoEL, decided to stop trying to build a full 3D model of everything (like a point cloud) and instead focus on the one thing event cameras are actually good at: lines.

Think of a room full of furniture. Even if the lighting is terrible or you are moving fast, the edges of a table, the corners of a window, and the lines of a door frame are still there. They are the "skeleton" of the room. RoEL is a system designed to find these skeletons and build a clean 3D map out of them.

Here is how it works, using a simple analogy:

1. The "Multi-Window" Detective (Finding the Lines)

Since the event camera's "flashes" are messy, looking at them all at once is confusing.

• The Analogy: Imagine trying to find a specific person in a crowded, foggy room. If you look at the whole room for 10 seconds, the person might move and blur. If you look for 1 second, you might miss them.
• RoEL's Trick: It looks at the room through many different "time windows" simultaneously. It creates several different "sketches" of the scene using different time intervals. It then combines all these sketches. If a line appears in any of the sketches, it keeps it. This ensures it doesn't miss any important edges, even if they are faint in some views.

2. The "Space-Time" Filter (Cleaning the Noise)

Once it has a bunch of candidate lines, some are real, and some are just random noise (false alarms).

• The Analogy: Imagine you have a pile of gold nuggets mixed with rocks. You want to separate them.
• RoEL's Trick: It uses a technique called Space-Time Plane Fitting. It treats the 3D space and the time dimension as a single sheet of paper. Real lines move smoothly across this sheet (like a train on a track). Random noise is scattered everywhere like confetti. RoEL finds the "smooth tracks" and throws away the confetti. It refines the lines to be perfectly straight and connects them to the specific flashes that created them.

3. The "3D Geometry" Brain (Building the Map)

Now that it has clean 2D lines from different angles, it needs to build the 3D map.

• The Analogy: Imagine two people standing in different spots, each holding a laser pointer at a wall. If they both point at the same spot, you know where the spot is. But if they are pointing at a long line, it's harder to figure out exactly where that line is in 3D space without getting confused.
• RoEL's Trick: Most systems try to project the 3D line back onto a 2D image to check if it's right (like looking at a shadow). But shadows can be misleading. RoEL uses a fancy math concept called Grassmann Distance.
  • Simple version: Instead of looking at the shadow, it measures the "angle" between the 3D line and the camera directly in 3D space. This prevents the system from getting confused about how far away the line is, ensuring the map is geometrically perfect.

4. The "Cross-Modal" Superpower (Using the Map for Other Things)

The final result is a 3D Line Map. It is incredibly compact (small file size) and very accurate.

• The Analogy: Because this map is so clean and structured, it can talk to other systems.
  • Registration: You can take this event-based map and snap it perfectly onto a standard 3D map made from a regular camera (like aligning a puzzle piece).
  • Localization: You can take a panoramic photo of a room and instantly figure out exactly where the camera is, just by matching the lines in the photo to the lines in the RoEL map.

Why This Matters

• Robustness: It works when other systems fail (in the dark, when moving super fast, or when the image is blurry).
• Efficiency: It uses very little memory because it only stores lines, not millions of points.
• Practicality: It proves that event cameras can be used for real-world robotics and navigation, not just in labs.

In a nutshell: RoEL takes the chaotic, flickering data of an event camera, filters out the noise by focusing on the "skeleton" of the room (lines), uses advanced math to build a perfect 3D structure, and creates a map that is so clean it can be used to navigate robots or align different types of cameras. It turns a messy stream of flashes into a reliable blueprint.

Technical Summary

1. Problem Statement

Event cameras offer high temporal resolution, high dynamic range, and low latency, making them ideal for high-speed and extreme lighting scenarios. However, their output is sparse, asynchronous, and highly sensitive to noise and motion, leading to significant domain discrepancies compared to frame-based cameras.

• Limitations of Existing Methods:
  • Direct Methods: Approaches like EMVS accumulate all events to build 3D maps. While simple, they are inherently sensitive to noise, often resulting in degraded reconstruction quality and dense but noisy point clouds.
  • Indirect Methods: Feature-based approaches (using corners or lines) are more robust to noise but struggle with event data due to the difficulty of extracting stable features from sparse, non-uniform event streams. Existing line-based methods (e.g., EL-SLAM) often rely on direct depth estimation or fail to handle the unique noise characteristics of events effectively.
• Core Challenge: How to construct a compact, noise-robust 3D line map from monocular event data while simultaneously refining camera poses, without relying on additional sensors or suffering from the instability of current feature extraction techniques.

2. Methodology: RoEL Pipeline

The authors propose RoEL (Robust Event-based 3D Line Reconstruction), a two-stage pipeline designed specifically for event data.

Stage 1: Robust Correspondence Search

This stage focuses on extracting reliable 2D line tracks and associating them with supporting events.

• Multi-Window, Multi-Representation (MWMR) Detection: To address the ambiguity of converting events to images (e.g., choosing the right temporal window), the method generates multiple "frames" using different temporal windows and accumulation strategies (Binary and Timestamp images). A lightweight 2D line detector is applied to all, and candidates are aggregated to minimize false negatives.
• Detection-Guided Space-Time Plane Fitting: Instead of treating events as a raw cloud, the method uses the initially detected 2D lines to guide a RANSAC-based plane fitting process in the $(x, y, t)$ space-time volume.
  • Events near a 2D line are selected as candidates.
  • A plane is fitted to these events.
  • Refinement: The intersection of the fitted plane with a constant-time slice yields a refined 2D line.
  • Association: Only "inlier" events (those close to the fitted plane) are associated with the line, effectively filtering noise and creating a compact representation.
• Hybrid Matching: A two-stage matching strategy establishes correspondences across time:
  • Local Matching: Mutual nearest neighbor search between adjacent frames for short-term consistency.
  • Global Matching: Uses a modality-invariant point matcher over longer intervals to resolve flickering or motion variations, merging tracks that correspond under the global model.

Stage 2: 3D Line Reconstruction & Optimization

This stage reconstructs 3D geometry and refines poses using geometric cost functions defined directly in 3D space.

• Triangulation: Initial 3D lines are estimated via RANSAC-based triangulation of multi-view 2D lines.
• Grassmann Distance Cost Functions: Unlike traditional methods that project 3D lines to 2D (losing depth information and introducing ambiguity), RoEL defines cost functions using the geodesic distance on the Grassmann manifold.
  • 2D Line-3D Line Cost: Measures the geometric consistency between a 3D line and its back-projected 2D observation plane.
  • Event-3D Line Cost: Measures the consistency between a 3D line and the rays of associated inlier events.
• Optimization: The system jointly optimizes 3D line parameters and camera poses by minimizing a weighted sum of these Grassmann costs.
  • Parameterization: To avoid over-parameterization and instability, 3D infinite lines are represented using a minimal orthonormal representation (4-DoF) during optimization, then converted back to Plücker coordinates for processing.
  • Trimming: Final 3D line segments are obtained by trimming the optimized infinite lines based on 2D observations.

3. Key Contributions

1. First Indirect Monocular Event Line Mapping: The first pipeline to perform 3D line mapping for monocular event data without relying on additional sensors (like stereo or RGB cameras).
2. Event-Specific Robustness Techniques: Introduction of MWMR detection and detection-guided space-time plane fitting to handle the sparsity and noise of event streams, enabling stable feature extraction.
3. Grassmann-Based Geometric Optimization: A novel formulation using Grassmann manifold distances for 3D reconstruction. This avoids the depth ambiguity and projection errors of traditional reprojection costs, allowing for accurate joint optimization of lines and poses.
4. Cross-Modal Applicability: The resulting 3D line maps serve as compact, mid-level representations that can be effectively used for cross-modal tasks, such as registering event maps to RGB-D point clouds or localizing against panoramic RGB images.

4. Experimental Results

The method was evaluated on synthetic (Replica, I2-SLAM) and real-world datasets (TUM-VIE, VECtor).

• Reconstruction Quality:
  • Accuracy & IoU: RoEL significantly outperforms baselines (EMVS, EL-SLAM, LIMAP) in Accuracy and Intersection over Union (IoU) metrics. On the Replica dataset, it achieved an average IoU@20 of 0.137, compared to 0.063 for EL-SLAM and 0.073 for LIMAP.
  • Compactness: Despite using only ~2,500 line segments on average, RoEL achieves higher geometric fidelity than EMVS, which uses nearly 900,000 noisy points.
  • Noise Robustness: Qualitative results show RoEL reconstructs clean structures (e.g., ceiling boundaries, furniture edges) where direct methods produce noisy clouds and other line methods fail to recover details.
• Pose Refinement:
  • Joint optimization using Grassmann costs significantly reduced Absolute Trajectory Error (ATE). On synthetic data with noisy poses, RoEL reduced ATE from 33.25m (LIMAP) to 6.92m.
• Cross-Modal Applications:
  • Registration: RoEL maps registered to RGB-D point clouds with lower rotation (0.714°) and translation (0.019m) errors than baselines.
  • Panoramic Localization: In localization tasks against RGB panoramas, RoEL achieved an 81.5% success rate (within 0.3m/15°), significantly outperforming EL-SLAM (30.5%) and LIMAP (61.0%).
• Challenging Conditions:
  • Under high-speed motion blur and extreme underexposure (where RGB cameras fail), RoEL successfully reconstructed scene structures that were completely lost in frame-based methods.

5. Significance

• Practical Deployment: RoEL demonstrates that event-based perception can be robust and practical for real-world robotics, particularly in dynamic or low-light environments where traditional cameras fail.
• Theoretical Advancement: By leveraging the Grassmann manifold for 3D line optimization, the paper provides a mathematically sound framework that preserves 3D geometric information, setting a new standard for indirect event-based SLAM.
• Versatility: The compact line maps serve as a universal intermediate representation, bridging the gap between event data and other modalities (RGB, Point Clouds), facilitating multi-sensor fusion and downstream tasks like localization and registration.

In conclusion, RoEL establishes a new state-of-the-art for event-based 3D mapping by combining robust feature extraction tailored to event characteristics with a geometrically rigorous optimization framework, proving that line-based representations are a superior choice for event cameras in man-made environments.