Technical Summary: Evita – Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing

1. Problem Statement

The integration of standard RGB frames with asynchronous event streams has become a critical paradigm for robust perception in degraded environments (e.g., high motion, extreme illumination). However, existing approaches face three fundamental bottlenecks:

1. Computational Redundancy: Prevailing methods rely on decoupled dual-encoder architectures, effectively doubling computational overhead and parameter counts.
2. Geometric Misalignment: Asynchronous sensor delays and spatial parallax cause implicit geometric misalignments between RGB and event data, which dual-stream architectures often ignore during feature aggregation.
3. Cross-Spectral Aliasing: The representational divide between dense absolute intensity grids (RGB) and sparse kinematic spikes (events) leads to severe semantic aliasing and high-frequency noise when simple spatial aggregation is applied.

While unified backbones have succeeded in RGB-Depth and RGB-Thermal domains, they fail in the RGB-Event domain because they treat heterogeneous modalities as homogeneous tokens, overlooking the extreme sparsity and asynchrony of event data.

2. Methodology: The Evita Framework

The authors propose Evita, the first unified backbone specifically engineered for dense RGB-Event parsing. Unlike prior works that fuse features at early or late stages, Evita embeds cross-modal interactions directly into every encoder layer through a symmetric, intertwined hierarchical architecture.

2.1 Core Architectural Components

Each Evita block integrates three pivotal co-learning modules:

• Geometric Parallax Rectification: To address spatial discrepancies without explicit depth calibration, the authors introduce a non-parametric, cross-modal deformable alignment operator. This module dynamically generates spatial offsets and modulation masks based on the concatenation of RGB and event features, ensuring kinematic edges are accurately anchored to photometric boundaries.
• Harmonic Spectral Resonance: To overcome modality heterogeneity and avoid spatial noise artifacts, texture transfer is executed exclusively in the complex frequency domain. Using Discrete Fourier Transforms (DFT), the method decouples amplitude and phase. It modulates the RGB amplitude spectrum with event-derived amplitudes (gated by a learned mechanism) while preserving the RGB phase structure, effectively injecting invariant high-frequency change matrices without introducing spatial aliasing.
• Transient Global Routing: This module formulates an asymmetric cross-modal attention mechanism. Kinematic event features serve as dynamic queries, while RGB features provide keys and values. Crucially, an explicit "transient prior" derived from the event stream is added to the attention logits, allowing the network to actively route macroscopic photometric context toward motion boundaries while filtering static background noise.

2.2 Pretraining Strategy and N-ImageNetV2

To ensure robust feature extraction and decouple representations from specific event encodings, the authors introduce N-ImageNetV2, a large-scale dataset of over 1.2 million strictly aligned RGB-event pairs.

• Alignment Pipeline: A semantic-guided registration pipeline (Algorithm 1) maps sparse events to Surface of Active Events (SAE), extracts keypoints via SuperPoint, and matches them with LightGlue. An iterative adaptive relaxation strategy estimates robust homography matrices to warp event streams into the RGB coordinate space.
• Stochastic Pretraining: During pretraining, the network is exposed to a hybrid training objective where 60% of pairs are strictly aligned, and 40% are intentionally misaligned via stochastic spatial jitter. This forces the network to learn robust invariant structural features rather than overfitting to static correspondences.
• Representation Mixing: The architecture accepts randomly sampled event representations (e.g., Binary, Voxel, SAE) from a repository during pretraining, enabling seamless accommodation of arbitrary event formats during downstream fine-tuning.

3. Key Contributions

1. Unified Backbone (Evita): The first unified architecture for dense RGB-Event parsing that eliminates the parameter inflation of decoupled paradigms by embedding cross-modal interactions in every encoder layer.
2. Novel Co-Learning Modules: The introduction of Geometric Parallax Rectification, Harmonic Spectral Resonance, and Transient Global Routing to address geometric parallax, cross-spectral aliasing, and dynamic context routing, respectively.
3. N-ImageNetV2 and Pretraining Protocol: A new dataset with strictly aligned pairs and a stochastic mixing protocol that bridges the distribution shift between absolute intensity and relative contrast, enabling format-agnostic downstream performance.
4. State-of-the-Art Performance: Establishment of new metrics across multiple benchmarks with a superior accuracy-latency trade-off.

4. Experimental Results

Evaluations were conducted on the DELIVER, DDD17, and DSEC benchmarks.

• DELIVER Benchmark: The flagship Evita-L achieved a new state-of-the-art 59.57% mIoU. Notably, it surpassed the strong baseline CMNeXt-B4 (58.87%) while utilizing only 38% of its parameters and approximately half the FLOPs. The lightweight Evita-S achieved 56.79% mIoU with only 21.9M parameters.
• DDD17 and DSEC Benchmarks: Evita-L set new records with 80.12% mIoU on DDD17 and 76.80% mIoU on DSEC. On DSEC, Evita-L required only 82.7 GFLOPs, nearly half the computational burden of MambaSeg (159.5 GFLOPs), while delivering a 2.0% absolute accuracy improvement.
• Efficiency: Evita-S operates at 22.4ms latency on an RTX 3090, significantly outperforming heavier models like OmniSegmentor and CMNeXt in real-time edge deployment scenarios.
• Robustness: Under simulated spatial misalignments (up to 48 pixels), Evita-L demonstrated exceptional resilience with a degradation of only 0.18% mIoU, compared to drops of >1.0% in competing unified or decoupled models.
• Generalization: The pretraining protocol successfully transferred to RGB-Thermal (MFNet) and RGB-LiDAR (KITTI-360) tasks, yielding performance gains over RGB-only initialization, suggesting the learned priors are fundamental to cross-modal fusion.

5. Significance and Claims

The paper claims that Evita fundamentally bridges the "extreme representational divide" between dense intensity grids and sparse kinematic spikes. By explicitly rectifying geometric parallax and integrating cross-spectral harmonics, the framework establishes a robust structural foundation for multimodal perception.

The authors assert that their approach:

• Eliminates Computational Redundancy: By moving away from dual-stream inference, Evita achieves optimal accuracy-latency equilibrium.
• Ensures Format Agnosticism: Through stochastic representation mixing, the network can handle arbitrary event encodings without retraining.
• Provides Resilient Perception: The learned harmonic-geometric priors enable robust scene understanding in unconstrained environments, particularly under adverse weather and motion blur where standard RGB sensors fail.

The work positions Evita not just as a segmentation model, but as a foundational step toward a universally agnostic multimodal architecture capable of handling diverse sensor modalities through unified geometric and spectral learning.