Technical Summary: Neurosim and Cortex

Problem Statement

The development of algorithms for neuromorphic perception, particularly using event-based cameras, faces significant hurdles due to the unique nature of event data. Unlike standard RGB images, event cameras produce asynchronous streams of intensity changes (4-tuples of timestamp, coordinates, and polarity), making traditional computer vision approaches inapplicable and supervised learning annotation extremely difficult. Existing solutions rely heavily on cross-modal supervision (e.g., transferring labels from RGB or LiDAR), which requires precise synchronization and calibration. Furthermore, the largest existing multi-sensor event datasets (e.g., M3ED) are limited in size (less than four hours of data).

Current simulators fail to adequately address these challenges. Many rely on low-frame-rate RGB rendering (e.g., 30 FPS) to approximate events, leading to temporal aliasing and significant artifacts. Others generate high-frame-rate RGB data, save it to disk, and post-process it to create events, a process that is prohibitively slow and storage-intensive (consuming terabytes for hours of simulation). Additionally, there is a lack of systems capable of feeding high-throughput, multi-modal sensor data (events, RGB, depth, IMU) directly into deep learning pipelines without disk I/O bottlenecks, which is essential for training foundation models and testing closed-loop control at the limits of hardware agility.

Methodology

The authors present Neurosim, a fast, real-time, high-performance simulator, and Cortex, a ZeroMQ-based communication library, designed to bridge the gap between simulation and machine learning workflows.

Neurosim Architecture

Neurosim is built on four design principles: modularity, low-latency, real-time streaming, and Pythonic accessibility.

1. Rendering and Event Generation: The system utilizes Habitat-Sim for high-fidelity scene rendering, capable of achieving ~3000 FPS for indoor scenes. Event simulation is performed on-the-fly by tracking the intensity state of each pixel. When the intensity change exceeds a threshold, an event is triggered.
2. GPU Optimization: To overcome the latency of CPU-based event generation, Neurosim implements a custom CUDA kernel. This kernel processes pixels in parallel and aggregates events at the warp level. By using shared memory (SRAM) for intermediate reductions and minimizing global atomic operations (reducing them from one per event to one per warp), the system achieves a massive throughput reduction in atomic contention. This allows event generation rates of >31 kHz for VGA and >23 kHz for HD frames.
3. Dynamics and Sensors: The simulator integrates RotorPy for multirotor dynamics, simulating aerodynamics, drag, and inertial sensors at >20 kHz on a single CPU core. It supports closed-loop control where sensor data and control commands are exchanged with minimal latency.

Cortex Communication Interface

Cortex is a high-performance, low-latency communication library designed to connect Neurosim with downstream consumers (controllers, trainers, visualizers).

1. Zero-Copy and Native Support: Built on ZeroMQ, Cortex supports native serialization for NumPy arrays and PyTorch tensors, enabling zero-copy data transfer between processes. This eliminates the need for disk I/O or excessive buffering.
2. Dynamic Discovery: It features a lightweight discovery daemon for dynamic node registration and topic management, allowing flexible peer-to-peer topologies without a centralized broker.
3. ROS 2 Integration: A bidirectional bridge allows users to leverage the ROS 2 ecosystem for control and state estimation while using Cortex for high-bandwidth data streams within Neurosim.

Key Contributions

1. High-Performance Event Simulation: Neurosim achieves event simulation speeds up to 2700 FPS on a single desktop GPU (Nvidia 4090), which is approximately 10× faster than existing GPU-based methods and 55–121× faster than CPU-based implementations.
2. Real-Time Streaming Pipeline: The integration of Neurosim and Cortex enables a "disk-less" training pipeline. Multi-modal data (events, RGB, depth, IMU) is streamed directly to deep learning training pipelines in Python-native formats, supporting online training of foundation models.
3. Closed-Loop Capability: The system supports closed-loop experiments at the extremes of hardware performance (e.g., agile flips, high-speed obstacle avoidance) with latencies as low as 0.7 ms, enabling the testing of reactive algorithms that would be too risky to perform on physical hardware.
4. Modular and Extensible Design: The architecture allows for the swapping of rendering backends, dynamics models, and controllers, facilitating diverse research in embodied perception.

Results

The paper benchmarks Neurosim and Cortex against existing tools (ESIM, VID2E, AirSim, PyTorch implementations):

• Latency: Neurosim's event simulator backend processes a single intensity frame update in roughly 0.37 ms (VGA), significantly outperforming competitors.
• Throughput: In full simulation steps (rendering + dynamics + event simulation), Neurosim achieves ~2300 FPS for VGA sensors. Other backends bottleneck the pipeline, achieving only 200–1200 FPS.
• Efficiency: Neurosim's event simulation consumes only ~8% of the total simulation step time, compared to 40–90% for other simulators.
• Communication: Cortex achieves message rates of 100+ kHz for small payloads and maintains 250 Hz for large 1080p images, with throughput saturating hardware limits at 7 GB/s.
• Applications Demonstrated:
  • Online Training: Successfully trained an event-based monocular depth estimation network (using F3 representation) completely online, consuming ~8.5M events/second without disk storage.
  • Reinforcement Learning: Trained a PPO-based policy to stabilize a quadrotor from random high-velocity initial conditions using only event camera observations, demonstrating the system's ability to handle complex, dynamic control tasks.

Significance

The authors claim that Neurosim and Cortex open new avenues for building and testing neuromorphic algorithms by addressing the critical bottlenecks of data generation, storage, and latency. The system enables:

• Scalable Learning: Training on diverse, time-synchronized multi-modal data without the logistical burden of managing terabytes of static datasets.
• Privileged Supervision: Providing high-rate ground truth (metric depth, pose, cross-modal views) for self-supervised and reinforcement learning.
• Hardware-Realistic Testing: Allowing researchers to push algorithms to the limits of hardware agility (e.g., flips, high-speed navigation) in a safe, simulated environment.
• Controlled Studies: Facilitating precise investigations into sensor timing, clock skew, and asynchronous sampling.

The paper concludes that these tools are essential for advancing embodied autonomy, particularly for applications requiring fast, robust perception in power-constrained and dynamic environments. Future work includes hardware-in-the-loop experiments and support for Unreal Engine 5 as a rendering backend.