🔬 optics

Frequency-domain Event-based Imaging for Selective Surveillance

This paper introduces FRIES, a neuromorphic framework that leverages frequency-domain analysis of event-based camera data to selectively detect and visualize man-made objects, such as drones, by identifying their characteristic periodic signatures while suppressing background noise and unstructured motion.

Original authors: Megan Birch, James Rick, Adrish Kar, Jason Zutty, Joseph L. Greene

Published 2026-05-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Megan Birch, James Rick, Adrish Kar, Jason Zutty, Joseph L. Greene

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to listen to a specific violin soloist in a crowded, noisy concert hall. The audience is clapping, people are shuffling their feet, and the wind is rattling the windows. If you just try to "see" the whole room at once (like a standard camera), the violinist might get lost in the blur of movement.

This paper introduces a new way to "listen" to the visual world using a special kind of camera called an Event-Based Camera (EBC). Instead of taking full pictures like a regular camera, this camera only records tiny "events" whenever something changes brightness at a specific pixel. It's like having a microphone that only clicks when a sound happens, ignoring silence.

However, this creates a new problem: the camera produces a chaotic, scattered stream of clicks. It's hard to tell which clicks belong to the violinist (the target) and which belong to the shuffling audience (the background noise).

To solve this, the researchers created a system called FRIES (Frequency Rate Information for Event-Space) and a visualization tool called RTS (Resonant Time Surface). Here is how they work, using simple analogies:

1. The Problem: The "Noisy Room"

Standard surveillance cameras struggle when things move fast, when there is low light, or when the background is busy (like trees blowing in the wind). They often confuse a bird with a drone because they look similar in a static picture.
Event cameras are great at ignoring static backgrounds, but their output is so sparse and fast that traditional software can't easily find the "needle in the haystack."

2. The Solution: FRIES (The Frequency Detective)

The researchers realized that man-made objects (like drone rotors or mechanical choppers) move in a very specific, rhythmic pattern. They vibrate or spin at a steady frequency (like a drumbeat). Natural things (like leaves or clouds) move more randomly and slowly.

FRIES acts like a detective that listens for that specific drumbeat:

Step 1: The Gatekeeper (Time Gating)
Imagine a bouncer at a club who only lets people in if they arrive at a specific rhythm. FRIES filters out events that happen too fast (noise) or too slow (background drift). It keeps only the events that fit the "rhythm" of a mechanical object.
Step 2: The Crowd Gatherer (Clustering)
Once the "rhythmic" events are filtered, FRIES groups them together. If a bunch of rhythmic clicks are happening in one spot, it marks that spot as a "Region of Interest" (a potential target). It ignores the rest of the room.
Step 3: The Frequency Analyzer (Spectral Analysis)
For each group, FRIES asks: "What is the exact beat of this group?" It uses math to find the dominant frequency. If a group of events is beating at 110 times per second, FRIES knows that's likely a mechanical chopper. If it's 75 times per second, it's likely a drone.
Step 4: The Filter (RTS Visualization)
This is the final magic trick. The Resonant Time Surface (RTS) is like a spotlight that only shines on things moving in sync with the beat you found.
- If an event matches the frequency (in-sync), the spotlight shines bright.
- If an event is out of sync (like a leaf blowing in the wind), the spotlight dims or turns it off.
- Result: You see a clear, glowing image of just the drone, while the messy background disappears.

3. What They Tested

The team tested this in two scenarios:

Indoors: They spun a mechanical chopper and a drone in a lab with a fake tree in the background. FRIES successfully found the chopper spinning at 110Hz and the drone rotors spinning at different speeds, ignoring the moving tree.
Outdoors: They flew a drone in front of a real, windy tree line. A regular camera saw a blurry mess where the drone was hard to see. FRIES, however, filtered out the wind-blown leaves and highlighted the drone's rotors clearly.

4. The Catch (Limitations)

The paper is honest about where the system still needs work:

False Alarms: In the wild, windy outdoors, the system sometimes got confused by random movements in the trees, thinking they were targets.
Stability: If the drone wobbles or changes speed too quickly, the "beat" gets messy, and the system might lose track of it.
Tuning: Right now, the system needs a human to tweak settings for different environments. It isn't fully "self-driving" yet.

The Bottom Line

This paper proposes a new way to watch the world: instead of looking for shapes (like a bird vs. a plane), it looks for rhythms. By treating moving objects like musical instruments with a specific beat, FRIES can pick out man-made machines from a chaotic background. It's not a replacement for regular cameras yet, but it's a powerful new tool that could help surveillance systems spot targets that are otherwise invisible to the naked eye.

Technical Summary: Frequency-domain Event-based Imaging for Selective Surveillance

Problem Statement

Existing surveillance systems face significant performance degradation when operating under rapid motion, low-contrast conditions, complex backgrounds, or degraded environments (e.g., fog, optical turbulence). These limitations create operational gaps in Automatic Target Recognition (ATR) pipelines, particularly for small Unmanned Aerial Systems (UAS) where distinguishing targets from confusers like birds is difficult. While Event-based Cameras (EBCs) offer high temporal resolution (microsecond-level) and high dynamic range by encoding pixel-level radiance changes, their asynchronous and sparse output presents challenges. Traditional image processing pipelines are incompatible with event streams, often requiring the reconstruction of intensity frames which diminishes the contrast-driven benefits of EBCs. Furthermore, existing event-space methods (e.g., time surfaces, neural networks) rely on spatial or temporal clustering cues, assuming that clustered events arise from similar structures. However, they often fail to leverage the periodic spectral signatures inherent in man-made objects (e.g., rotor rotation, mechanical vibrations) to distinguish them from unstructured background noise. Additionally, current detection performance is sensitive to user-defined contrast thresholds that require precise, scene-dependent calibration.

Methodology: FRIES and RTS

To address these gaps, the authors introduce Frequency Rate Information for Event-Space (FRIES), a neuromorphic processing framework designed to discriminate man-made targets by detecting periodicity in pixel-level events. The framework operates in four stages:

Temporal Gating: The raw event stream is separated by polarity to prevent cancellation at oscillating edges. Events are batched into fixed intervals (10ms). A time gate is applied to each pixel, retaining events only if the interval between the current event and the previous event falls within a specific range ( $t_{min}$ to $t_{max}$ ). This suppresses high-frequency sensor noise and slow background changes while preserving the frequency band of interest (e.g., 50–500 Hz for drone rotors).
Spatial Clustering: Retained events are summed per pixel to create a 2D activity map. An adaptive threshold, set to the median activity score of the map, filters out background noise. This approach assumes targets are sparse and produce higher event rates than the background. The thresholded map is downsampled, and DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is applied to group spatially contiguous high-activity pixels into Regions of Interest (ROIs). This step reduces computational load by limiting spectral analysis to candidate areas.
Spectral Analysis: For each ROI, events are retrieved from the unfiltered stream and arranged into per-pixel time series. A discrete Fourier Transform (DFT) is applied to each pixel's time series to generate a Power Spectral Density (PSD) map. The dominant frequency (peak power) is extracted per pixel, and each ROI is assigned a single dominant frequency based on the most common peak across its constituent pixels.
Resonant Time Surface (RTS) Visualization: RTS is a frequency-selective visualization tool that weights incoming events based on their phase coherence with the extracted target frequency. Using a time-localizing Gaussian filter convolved with a comb function at the target frequency ( $T = 1/f_t$ ), RTS assigns weights to events. Events in-phase with the target frequency are maximally weighted, while out-of-phase events are suppressed or penalized. This creates a surface that selectively enhances synchronized content and rejects clutter.

Key Contributions

FRIES Framework: A novel processing pipeline that leverages the high temporal resolution of EBCs to detect man-made objects via their spectral signatures rather than relying solely on spatial or temporal clustering.
Adaptive Thresholding: The use of a median-based activity threshold allows the system to operate robustly across different sensors and environments without manual retuning or precise knowledge of noise models.
RTS Visualization: A method to selectively visualize targets by weighting events based on phase coherence with a specific frequency, enabling the isolation of periodic motion from unstructured background noise.
Experimental Validation: Demonstration of the framework in both controlled indoor environments (mechanical chopper and drone rotors) and uncontrolled outdoor environments (hovering drone against a treeline).

Results

The framework was tested using a Prophesee EVK4 event-based camera in two scenarios:

Indoor Laboratory Experiment:
- FRIES successfully recovered the rotational frequency of a mechanical chopper at 110 Hz, matching the manufacturer's specifications.
- For a multi-rotor drone, four distinct rotor ROIs exhibited dominant frequencies between 129–451 Hz.
- The system effectively separated these high-frequency targets from the lower-frequency background activity generated by a manually agitated tree.
- RTS visualization successfully isolated the drone rotor at 391 Hz, suppressing unstructured background events that blurred the target in standard time surface representations.
Outdoor Field Experiment:
- A drone hovering in front of a wind-driven treeline was detected against a complex, low-contrast background.
- Time gating significantly reduced false positives caused by foliage motion and noise.
- Spectral analysis identified a dominant frequency near 75 Hz for the drone, while background tree motion exhibited lower frequencies (2–27 Hz).
- RTS visualization improved the distinction between the drone and background by approximately 4x in terms of event density ratio compared to standard time surfaces.
- However, the outdoor results showed intermittent false positives from background clutter, indicating that while the frequency separation works, the system's selectivity in uncontrolled environments requires further tuning.

Significance and Claims

The paper posits that frequency-domain event processing is a promising front-end for selective surveillance in neuromorphic pipelines. The authors claim that FRIES and RTS offer a complementary approach to conventional framing cameras, particularly in scenarios requiring high temporal resolution and spectral discrimination.

Complementarity: The authors modestly claim that EBCs may prove valuable as a complementary system to conventional cameras rather than a direct alternative, as EBCs trade salient spatial features for temporal resolution.
Selective Surveillance: The primary significance lies in the ability to discriminate structured object signatures (periodic motion) from unstructured background noise without relying on high-contrast spatial features.
Robustness: The adaptive nature of the framework (median-based thresholding) allows operation across varying sensors and environments without extensive calibration.
Limitations: The paper acknowledges that the current design relies on heuristic parameter selection (e.g., time gating windows) and that the outdoor results, while promising, are not yet reliable for operational use due to sporadic false positives. The authors emphasize the need for further tuning, adaptive thresholding, and algorithmic improvements (e.g., incremental clustering) to transition from a proof-of-concept to a deployable system.