💻 computer science

eNavi: Event-based Imitation Policies for Low-Light Indoor Mobile Robot Navigation

This paper introduces eNavi, a multimodal dataset and a late-fusion RGB-Event imitation learning policy that significantly enhances indoor mobile robot navigation robustness and accuracy, particularly in low-light conditions where conventional RGB-based approaches fail.

Original authors: Prithvi Jai Ramesh, Kaustav Chanda, Krishna Vinod, Joseph Raj Vishal, Yezhou Yang, Bharatesh Chakravarthi

Published 2026-03-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Prithvi Jai Ramesh, Kaustav Chanda, Krishna Vinod, Joseph Raj Vishal, Yezhou Yang, Bharatesh Chakravarthi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to walk through a dark, crowded room while holding a tray of drinks. If you rely only on your eyes (like a standard camera), the moment the lights flicker or you move too fast, everything blurs into a mess. You might trip or spill your drink.

Now, imagine you have a superpower: instead of seeing "pictures," you only notice changes. If a shadow moves, if a person walks by, if a light flickers—you instantly feel that shift, even in pitch blackness. This is how Event Cameras work. They don't take photos; they take "snapshots of change" at lightning speed.

This paper, titled "eNavi," is about teaching a robot to use this superpower to follow a person around a house, even when the lights are dim or the robot is moving fast.

Here is the story of how they did it, broken down into simple parts:

1. The Problem: The "Blurry Night" Dilemma

Standard robots use regular cameras (like your phone). These are great in a sunny park, but they struggle indoors:

Low Light: They get grainy and dark.
Fast Motion: They get blurry (motion blur).
Result: The robot gets confused and stops or crashes.

The researchers wanted to build a robot that could follow a human through a dark office or a messy living room without stumbling.

2. The Solution: A New "Robot Gym" (The Dataset)

To teach a robot, you need to show it examples of how to do it right. But there was no "gym" for robots using these special event cameras. So, the team built one.

The Setup: They put a standard camera and an event camera on a little robot (a TurtleBot).
The Training: A human drove the robot around a room, following a person, in both bright light and very dark light.
The Data: They recorded everything: the regular video, the "change-snapshots" (events), and exactly how the human steered the robot.
The Result: They created eNavi, the first-ever dataset that pairs these two types of vision with real steering commands. It's like a video game cheat code for robots: "Here is exactly what the robot should do in this situation."

3. The Brain: The "Bilingual" Robot

The researchers didn't just give the robot the data; they built a new "brain" (an AI model) to learn from it. They called it eNavi.

Think of the robot's brain as having two ears:

Ear A (The RGB Camera): Good at seeing colors and details when the lights are on.
Ear B (The Event Camera): Good at sensing movement and edges, even when it's pitch black.

The Magic Trick (Late Fusion):
Instead of forcing the robot to choose one ear, they built a Translator (a Transformer module) in the middle.

When the lights are bright, the robot listens mostly to Ear A (the color camera).
When the lights go out or the robot moves fast, Ear A goes deaf (blurry).
The Translator instantly switches focus to Ear B, which is still hearing perfectly because it only cares about movement.
The robot combines both inputs to make a decision. It's like having a bilingual friend who speaks English in the day and switches to a secret code at night so you never get lost.

4. The Results: The "Super-Listener" Wins

They tested three types of robots:

The "Day-Robot": Only uses the regular camera.
The "Night-Robot": Only uses the event camera.
The "Hybrid-Robot": Uses both (the eNavi model).

The Findings:

In the Dark: The "Day-Robot" got lost immediately. The "Night-Robot" did okay, but the "Hybrid-Robot" was the smoothest and most accurate.
Speed: The Hybrid-Robot learned faster. Because the event camera gives such clear signals about movement, the robot didn't have to guess as much.
Real-World Test: When they tested the robot in a dark room it had never seen before, the Hybrid-Robot kept following the person perfectly, while the others stumbled.

Why This Matters

This paper is a big step forward because it proves that robots don't need to be blind in the dark. By combining the "what it looks like" (RGB) with the "what is moving" (Events), we can build robots that are safe and reliable in our messy, dimly lit homes.

In a nutshell:
The researchers built a new dataset and a smart robot brain that acts like a night-vision superhero. It uses a special camera that sees "motion" instead of "pictures," allowing it to follow people through dark rooms without tripping, something regular robot cameras just can't do.

1. Problem Statement

Indoor mobile robot navigation, particularly tasks like person-following, faces significant challenges in environments with clutter, dynamic motion, and variable lighting.

Limitations of RGB Cameras: Conventional frame-based RGB cameras suffer from motion blur, limited dynamic range, and severe performance degradation in low-light conditions. This leads to a loss of perceptual quality and control stability.
Gap in Event-Based Research: While event cameras (which capture asynchronous brightness changes with microsecond latency and high dynamic range) have advanced in perception tasks (SLAM, odometry), there is a lack of research on end-to-end control policies that leverage the asynchronous nature of event streams for real-world navigation.
Data Scarcity: There are no existing datasets that pair high-quality event streams with expert control actions (ground truth) in realistic indoor navigation settings, hindering the development of supervised imitation learning for this modality.

2. Methodology

The authors propose a comprehensive solution comprising a new dataset, a data processing pipeline, and a novel navigation architecture.

A. The eNavi Dataset

Platform: Built on a TurtleBot 2 equipped with an NVIDIA Jetson Orin Nano.
Sensors:
- Event Camera: Prophesee Metavision EVK4 (1280×720, 220µs latency).
- RGB Camera: Teledyne Blackfly S (1280×1024, 170 fps).
- Synchronization: Both cameras are mounted on a beam splitter for co-axial alignment and triggered by a hardware-level microcontroller (STM32F407) to ensure precise temporal alignment.
Collection Protocol:
- Collected over 2 hours across 175 episodes in three distinct indoor environments.
- Includes diverse trajectories (single-path and multi-path) and lighting conditions (Normal and Low-light).
- Ground Truth: Expert teleoperation via joystick, but control labels are reconstructed from wheel odometry (linear and angular velocity) rather than raw joystick signals to ensure physical accuracy and reduce noise.
Data Format: Synchronized tuples of $(I_k, E_k, a_k)$ stored in HDF5, where $I$ is the RGB frame, $E$ is the event frame, and $a$ is the action.

B. Multimodal Data Processing Pipeline

To handle the asynchronous nature of event data vs. the synchronous nature of RGB frames:

Temporal Alignment: RGB frame timestamps serve as the anchor. Events occurring between $t_{i-1}$ and $t_i$ are aggregated.
Event Representation: Raw events are converted into a dense 2-channel spatial histogram (separating ON/OFF polarities) to match the spatial dimensions of the RGB input.
Action Reconstruction: Control commands are aggregated over the same time window as the visual data, computing the mean linear ( $v$ ) and angular ( $\omega$ ) velocities from odometry data to create smooth training labels.

C. Navigation Policy Architecture (ENP-Fusion)

The proposed Event-based Navigation Policy (ENP) utilizes a Late-Fusion strategy:

Dual Encoders:
- RGB Encoder: A pre-trained MobileNetV3-Small (frozen weights) to extract spatial features.
- Event Encoder: A trainable MobileNetV3-Small to learn features specific to sparse, high-temporal-resolution event data.
Fusion Module: The feature vectors from both encoders are concatenated and processed by a Transformer-based self-attention block. This allows the model to adaptively weigh the contribution of RGB context vs. event dynamics based on the current lighting and motion conditions.
Control Head: A Multi-Layer Perceptron (MLP) predicts continuous differential-drive commands ( $v, \omega$ ).
Training: The model is trained via Behavioral Cloning (BC) using Mean Absolute Error (MAE) loss to minimize the discrepancy between predicted and expert actions.

3. Key Contributions

eNavi Dataset: The first real-world, indoor person-following dataset featuring synchronized raw event streams, RGB frames, and expert control actions across varying lighting conditions.
Processing Pipeline: A robust methodology for temporally aligning asynchronous event data with RGB frames and reconstructing high-fidelity odometry-based ground truth actions.
Novel Architecture: A late-fusion RGB-Event navigation policy using dual MobileNet encoders and a Transformer fusion module.
Comprehensive Evaluation: A systematic study of 12 model variations (combining 3 architectures, 2 path complexities, and 2 lighting regimes) demonstrating the efficacy of event data in low-light scenarios.

4. Experimental Results

The authors evaluated the policy across 12 training variations and tested for zero-shot generalization to low-light conditions.

Training Efficiency:
- Event-driven models (Event-only and Fusion) converged significantly faster (often stopping early between epochs 20–35) compared to RGB-only models, which often required the full 50 epochs.
- Fusion models achieved the lowest validation Mean Absolute Error (MAE). For example, in the complex Multi-Path/Mixed-Light setting, Fusion achieved an MAE of 0.0370 vs. 0.0707 for RGB-only.
Low-Light Robustness (Zero-Shot Generalization):
- Policies trained only on Normal-Light data were tested on Low-Light data.
- Simple Trajectories: RGB-only models performed adequately.
- Complex Trajectories: RGB-only models showed significant performance degradation. In contrast, Fusion models maintained robustness, with a much smaller increase in error when transitioning from Normal to Low light (Total MAE change: 0.0335 $\to$ 0.0467 for Fusion vs. 0.0463 $\to$ 0.0514 for RGB).
Trajectory Analysis: Visualizations confirmed that the Fusion policy maintained accurate linear and angular velocity predictions even when RGB features were degraded, successfully tracking high-speed segments and complex turns.

5. Significance and Future Directions

Bridging the Gap: This work successfully bridges the gap between event-based perception and closed-loop robot control, proving that event data is not just for perception but critical for robust control in challenging environments.
Real-World Applicability: By providing a real-world dataset and demonstrating superior performance in low-light conditions, the paper offers a viable path for deploying service robots in homes and offices where lighting is unpredictable.
Limitations & Future Work: The authors note that the current synchronized processing pipeline is computationally heavy for real-time edge inference. Future work should focus on asynchronous processing (handling data at native intervals) and advanced policy heads (e.g., diffusion-based) to improve real-time responsiveness.

In conclusion, eNavi establishes a new benchmark for event-based imitation learning, demonstrating that fusing event cameras with RGB cameras significantly enhances the robustness and generalization of indoor mobile robots, particularly in low-light scenarios where traditional vision fails.