Deep Learning-based Event Data Coding: A Joint Spatiotemporal and Polarity Solution

1. Problem Statement

Event cameras (neuromorphic vision sensors) capture visual data asynchronously as streams of events $(x, y, t, p)$, offering high temporal resolution, high dynamic range, and low latency. However, this generates massive data volumes, creating significant challenges for storage and transmission.

• Current Limitations: Existing solutions primarily rely on lossless coding to preserve data integrity for computer vision (CV) tasks, resulting in high bitrates.
• The Gap: While lossy coding is standard for human-centric media (images/video), it has been avoided for event data under the assumption that any distortion harms CV performance. Furthermore, existing Point Cloud Coding (PCC) approaches for event data typically represent events as two separate point clouds (one for positive polarity, one for negative), failing to exploit the correlation between the two polarities and missing opportunities for joint compression.
• Goal: Develop a lossy Deep Learning (DL)-based coding solution that achieves significant compression rates while maintaining or improving performance for downstream CV tasks (specifically classification).

2. Methodology: DL-JEC Architecture

The authors propose DL-JEC (Deep Learning-based Joint Event data Coding), a novel framework that treats event data as a single Point Cloud (PC) rather than two separate ones.

A. Data Representation

• Single PC Representation: Instead of splitting data by polarity, DL-JEC maps the event tuple $(x, y, t, p)$ to a single 3D point cloud where:
  • Geometry: $(x, y, z)$ corresponds to spatial coordinates $(x, y)$ and scaled time $t$ (where $z = t \times TSF$).
  • Attributes: The polarity $p$ (Positive/Negative) is treated as a binary attribute attached to the geometry.
• Preprocessing: Includes temporal scaling, voxelization, and duplicate removal (both same-polarity and different-polarity duplicates at the same voxel).

B. Coding Model

The architecture is an extension of the JPEG Pleno PCC standard but modified for joint coding:

• Input: A sparse tensor with two channels: Channel 1 for occupancy (geometry), Channel 2 for polarity.
• Encoder: Uses a Variational Autoencoder (VAE) with a hyperprior model. Unlike JPEG PCC (which codes geometry and attributes separately for scalability), DL-JEC fuses geometry and polarity features early in the latent space. This allows the model to exploit correlations between spatiotemporal data and polarity.
• Decoder: Reconstructs the occupancy probability and polarity probability for each voxel.
• Loss Function: A novel loss function balances rate ($R$) and total distortion ($D_{Total}$):
  $$Loss = D_{Total} + \lambda \times R$$
  Where $D_{Total}$ is a weighted sum of Geometry Distortion and Polarity Distortion, both calculated using Focal Loss.

C. Adaptive Binarization Strategies

Since the decoder outputs probabilities, a binarization step is required to reconstruct the final event stream. The paper introduces three distinct strategies optimized for different goals:

1. Count-optimized (CoB): Reconstructs the exact number of input events ($k = N_{input}$). Good for tasks sensitive to event density.
2. Quality-optimized (QuB): Selects $k$ to maximize reconstruction fidelity (measured by a new metric, PSNR E2E).
3. Classification-optimized (ClB): Selects $k$ to maximize the Top-1 classification accuracy of the downstream task. This strategy acts as a learned denoising filter, often discarding noisy events that hinder classification.

D. New Quality Metric: PSNR E2E

The paper proposes Peak Signal-to-Noise Ratio Event-to-Event Distance (PSNR E2E). Unlike standard PCC metrics (PSNR D1) that only consider geometry, PSNR E2E accounts for:

• Spatial and temporal distances.
• Polarity matching: It calculates the Mean Squared Error (MSE) between nearest neighbors of the same polarity in the reference and reconstructed sets.

3. Key Contributions

1. First Joint Single-PC Coding: The first DL-based solution to code event data as a single point cloud with polarity as an attribute, exploiting spatiotemporal-polarity correlations.
2. Task-Oriented Binarization: Novel adaptive strategies (CoB, QuB, ClB) that allow the codec to be tuned specifically for data fidelity or specific CV tasks (classification) without retraining the core model.
3. PSNR E2E Metric: A new quality metric tailored for lossy event data that considers both geometry and polarity.
4. Paradigm Shift: Demonstrates that lossy coding can outperform lossless coding in terms of rate while maintaining or improving classification accuracy.

4. Experimental Results

The solution was evaluated on two datasets: N-Caltech101 (object classification) and IBM DVS Gesture (gesture recognition), using the EST (Event Spike Tensor) classifier.

• Compression Performance (Rate-Distortion):

  • DL-JEC significantly outperforms MPEG G-PCC (lossy and lossless) and JPEG Pleno PCC (lossy) in Rate-Distortion (RD) curves when using the QuB strategy.
  • It achieves high reconstruction quality at bitrates far below the lossless G-PCC baseline.
  • Computational Efficiency: DL-JEC has lower inference times (encoding/decoding) compared to JPEG PCC.

• Classification Performance:

  • ClB Strategy: The Classification-optimized (ClB) strategy achieved the highest Top-1 and Top-5 accuracy, often surpassing the original uncompressed data and the voxelized baseline.
  • Reasoning: The ClB strategy effectively filters out noise and irrelevant events, providing a "cleaner" input to the classifier.
  • Generalization: Models trained on N-Caltech101 generalized well to the IBM DVS Gesture dataset, maintaining competitive performance against benchmarks.

• Comparison with Benchmarks:

  • DL-JEC consistently outperformed the "Two-PC" approach (JPEG PCC) in both compression efficiency and classification accuracy.
  • The gap was wider for CoB strategies, as the Two-PC approach generates duplicate events with conflicting polarities that must be removed, wasting bits.

5. Significance and Impact

• Redefining Event Coding: The paper challenges the dogma that event data must be losslessly coded. It proves that controlled lossy compression is not only viable but beneficial for machine vision tasks.
• Standardization Potential: The work aligns with the JPEG XE (Event Data) standardization efforts, providing a strong technical foundation for future lossy event coding standards.
• Efficiency: By enabling lower bitrates without sacrificing (and often improving) task performance, DL-JEC facilitates the deployment of event cameras in bandwidth-constrained applications like autonomous driving, robotics, and IoT.
• Future Directions: The authors suggest extending this framework to other CV tasks (e.g., semantic segmentation) and training on more diverse datasets to improve generalization across different event densities.