💻 computer science

Adaptive Event Stream Slicing for Open-Vocabulary Event-Based Object Detection via Vision-Language Knowledge Distillation

This paper proposes an adaptive event stream slicing framework for open-vocabulary object detection that bridges the modality gap between event cameras and vision-language models by using image-based CLIP knowledge distillation to guide a hybrid SNN-CNN student network, which adaptively segments event streams to preserve crucial temporal features while inheriting broad semantic understanding.

Original authors: Jinchang Zhang, Zijun Li, Jiakai Lin, Guoyu Lu

Published 2026-03-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jinchang Zhang, Zijun Li, Jiakai Lin, Guoyu Lu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to identify objects in a room, but instead of seeing a full, colorful photograph, you only see a chaotic stream of tiny, invisible sparks flying around whenever something moves. This is how Event Cameras work. They are super-fast, super-sensitive sensors that only "see" changes (like a car speeding by or a person walking), ignoring everything static. They are great for high-speed action and work in the dark, but they have a major flaw: they are blind to texture and color. They see a "spark" where a car is, but they don't know it's a "car" until they've been taught.

This paper introduces a new system to teach these spark-sensors to recognize any object they've never seen before (like a "fire hydrant" or a "dog") just by reading a text description. Here is how they did it, broken down into simple concepts:

1. The Problem: The "Language Barrier"

Think of Event Cameras as a person who only speaks "Spark-ese" (fast, chaotic, no color).
Think of CLIP (a famous AI that understands images and text) as a person who only speaks "Photo-English" (rich, detailed, colorful).

If you try to ask the Photo-English speaker to understand Spark-ese directly, they get confused. The paper says, "We can't just force the AI to learn Spark-ese from scratch because we don't have enough textbooks (data) for it."

2. The Solution: The "Teacher-Student" Tutoring System

The authors created a clever tutoring setup called Knowledge Distillation.

The Teacher: A super-smart AI (CLIP) that looks at normal photos and knows exactly what a "car" or a "streetlight" looks like.
The Student: The Event Camera AI that only sees sparks.

Instead of forcing the Student to learn from scratch, the Teacher shows the Student a photo of a car, explains what it is, and then points to the corresponding "sparks" in the event stream. The Student learns to say, "Ah, these specific sparks mean 'car' because my Teacher told me so." This bridges the gap between the two different ways of seeing the world.

3. The "Smart Slicer": Cutting the Cake at the Right Time

Event cameras produce a continuous, never-ending stream of data. To process it, you have to chop this stream into little chunks (like slicing a loaf of bread).

Old Way: Cut the bread every 10 seconds, no matter what.
- Result: If a car is moving slowly, you might cut the slice in half, losing the car. If a car is zooming by, you might get a slice with 100 cars in it, making it messy.
New Way (Adaptive Slicing): The paper uses a Spiking Neural Network (SNN)—which acts like a biological brain neuron. It waits until the "sparks" reach a certain critical point before it says, "Okay, that's a complete scene! Cut here!"
- The Analogy: Imagine a chef slicing a loaf of bread. Instead of using a timer, the chef waits until the bread is perfectly risen and ready. If the bread is still rising (slow motion), they wait. If it's exploding with activity (fast motion), they cut immediately. This ensures every slice contains the perfect amount of information.

4. The "Self-Correcting Feedback Loop"

How does the "Smart Slicer" know when to cut? It uses a Self-Supervised Feedback mechanism.

The Analogy: Imagine a student taking a test. If they cut the bread (segment the data) at the wrong time, the AI tries to identify the object and fails (gets a bad grade).
The system then says, "Hey, that cut was bad. Next time, wait a little longer."
If the cut was good and the AI recognized the object, it says, "Great job! Do that again."
Over time, the system learns exactly when to slice the data to get the best results, without needing a human to teach it.

5. The Grand Result: "Open-Vocabulary" Detection

Because the Student learned from the Teacher (who knows thousands of words), the Event Camera can now detect anything you describe, even if it was never in the training data.

Scenario: You tell the camera, "Find me a 'fire hydrant'."
Result: The camera scans the spark stream, matches the sparks to the "fire hydrant" description it learned from the Teacher, and says, "Found one!" even though it has never seen a fire hydrant before.

Summary

This paper is like teaching a blind, super-fast runner (the Event Camera) how to identify objects by having them shadow a sighted, slow runner (the Photo AI). The blind runner learns to recognize objects by feeling the "sparks" of movement, guided by the sighted runner's descriptions. They also invented a smart way to pause and slice the action at the perfect moment, ensuring no crucial details are lost.

Why it matters: This technology could allow self-driving cars to see clearly in blinding fog, heavy rain, or at night, identifying pedestrians and obstacles that traditional cameras might miss, all while using very little battery power.

1. Problem Statement

Event cameras offer significant advantages over traditional frame-based cameras, including high temporal resolution, low latency, and robustness to motion blur. However, they face two critical challenges in object detection:

Modality Gap & Lack of Texture: Event data lacks color and texture information, making it difficult to apply standard Vision-Language Models (VLMs) like CLIP, which are trained on RGB images. Directly transferring CLIP to event data is ineffective due to the distribution mismatch.
Closed-Set Limitations: Existing event-based detectors are typically trained on predefined categories (closed-set), failing to generalize to novel, unseen objects common in real-world scenarios.
Inefficient Event Slicing: Current methods often use fixed strategies (e.g., fixed time intervals or event counts) to segment raw event streams. These static approaches lead to information loss in low-speed motion or redundancy in high-speed motion, failing to capture optimal temporal features.

2. Methodology

The authors propose a framework that bridges the gap between event streams and VLMs through Knowledge Distillation and solves the segmentation issue via an Adaptive SNN-based Slicing mechanism.

A. Adaptive Event Stream Slicing (SNN Module)

Instead of fixed grouping, the paper introduces a Spiking Neural Network (SNN) to dynamically determine the optimal moments to slice the event stream.

Mechanism: The SNN acts as a dynamic trigger. When a neuron fires (spikes), it defines the end of an event group for processing.
Loss Functions for Optimization:
- Membrane Potential Driven Loss (Mem-Loss): Ensures the neuron's membrane potential reaches the firing threshold ( $V_{th}$ ) exactly at the target time step.
- Linear Incremental Constraint Loss (LIC-Loss): Prevents "premature firing" (the "hill effect") by enforcing a monotonic increase in membrane potential over time, ensuring stable and consistent segmentation.
- Self-Supervised Feedback Loss (SSF-Loss): A novel feedback loop where the downstream object detection loss ( $L_M$ ) directly guides the SNN's membrane potential. If a specific slicing time step leads to poor detection results, the SNN adjusts to avoid firing at that time, effectively learning the optimal segmentation strategy based on detection performance.

B. Vision-Language Knowledge Distillation

To enable open-vocabulary detection without large-scale event-text datasets, the framework transfers knowledge from a pretrained CLIP model (Teacher) to the event-based detector (Student).

Teacher-Student Setup: The frozen CLIP image encoder serves as the teacher, processing reconstructed image frames from the event stream. The event-based detector is the student.
Spatial Attention Distillation: To handle the sparsity of event data, a spatial attention mechanism is introduced. It generates attention maps from the teacher's features to highlight key regions and suppress noise in the student's event features. A contrastive loss aligns the event-region embeddings with the image-region embeddings.
Category-Agnostic Design: The detector uses a category-agnostic proposal module. Instead of predicting specific classes, it predicts generic bounding boxes and masks. Classification is performed by computing the cosine similarity between the region features and text embeddings generated by the frozen CLIP text encoder (e.g., "A photo of a car").

C. Inference

During inference, the model operates solely on event stream data. It does not require image frames, relying on the features learned during the distillation phase to classify objects into novel categories based on text prompts.

3. Key Contributions

First Open-Vocabulary Event Detector: The first framework to enable open-vocabulary object detection on event cameras directly using textual descriptions.
Adaptive Event Slicing: A novel SNN-based module that dynamically learns optimal segmentation points using self-supervised feedback, overcoming the limitations of fixed-time/event-count slicing.
Cross-Modal Knowledge Distillation: A robust method to transfer rich semantic knowledge from CLIP (image domain) to event data without requiring massive event-text datasets, utilizing spatial attention to bridge the modality gap.
Category-Agnostic Architecture: A design that decouples localization from classification, significantly improving generalization to unseen object categories.

4. Experimental Results

The method was evaluated on three datasets: NCAR (binary classification), DSEC (driving scenarios), and GEN1 (automotive detection).

Performance on Novel Categories: On the DSEC dataset, the model achieved 42.4% accuracy on "Large Vehicles" (a novel category not seen during training), outperforming models trained specifically on that category.
Zero-Shot Generalization:
- NCAR: Achieved 95.7% accuracy, surpassing state-of-the-art SNN and non-SNN methods.
- GEN1: Achieved 65.7% mAP50 and 38.3% mAP50:95 in a zero-shot setting, outperforming existing SNN-based detectors (e.g., CREST, SFOD) and image-based open-vocabulary detectors adapted for event data.
Ablation Studies:
- Removing the Adaptive Slicing (using fixed slicing) caused a significant drop in performance.
- Removing Knowledge Distillation resulted in a massive performance degradation, confirming the necessity of transferring CLIP's visual knowledge.
- The Self-Supervised Feedback Loss further refined the slicing strategy, boosting mAP50 from 0.621 to 0.657.

5. Significance

This work represents a paradigm shift in event-based vision by:

Overcoming Data Scarcity: It circumvents the need for large-scale, annotated event-text datasets by leveraging existing VLMs through distillation.
Enhancing Robustness: The adaptive slicing mechanism ensures that critical temporal information is preserved regardless of motion speed, addressing a fundamental weakness in current event processing.
Enabling Real-World Applicability: By enabling detection of unseen objects (e.g., new vehicle types, pedestrians in novel poses) using only event data, the system is better suited for autonomous driving and robotics in dynamic, unpredictable environments where traditional closed-set detectors fail.

In summary, the paper successfully integrates the temporal efficiency of SNNs with the semantic power of VLMs, creating a robust, generalizable framework for open-vocabulary object detection on event cameras.