🔬 applied physics

Event Topology-based Visual Microphone for Amplitude and Frequency Reconstruction

This paper presents an event topology-based visual microphone that leverages the Mapper algorithm and hierarchical density-based clustering to accurately reconstruct vibration amplitude and frequency from raw event streams without external illumination, enabling high-fidelity, passive sensing of single and multiple sound sources.

Original authors: Ryogo Niwa, Yoichi Ochiai, Tatsuki Fushimi

Published 2026-02-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ryogo Niwa, Yoichi Ochiai, Tatsuki Fushimi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The "Silent Photographer" That Hears with Its Eyes

Imagine you are in a room where someone is whispering, but you can't hear them. Now, imagine you have a special camera that doesn't just take pictures; it sees the air vibrating. If you point this camera at a speaker playing music, or even a person talking, the camera can "see" the sound waves moving the speaker's cone or the person's eardrum, and then translate those movements back into sound.

This is what the paper calls a "Visual Microphone." But the researchers in this paper have built a new, super-powered version of it using a special kind of camera called an Event Camera.

Here is the simple breakdown of how it works, using some everyday analogies.

1. The Problem: The Old Cameras Were Too Clumsy

Traditional cameras are like a flipbook artist. They take a full picture (a frame) 30 or 60 times a second. To see something move very fast (like a vibrating guitar string), you need to take thousands of pictures per second. But taking that many pictures requires:

Bright lights (like a camera flash that never turns off).
Huge amounts of data (which slows things down).
Expensive equipment (like a laser scanner).

If you try to use a normal camera to listen to a whisper, it's like trying to catch a hummingbird with a butterfly net made of concrete. It's too slow and too heavy.

2. The Solution: The "Event Camera" (The Motion-Sensitive Security Camera)

The researchers used an Event Camera. Think of this camera not as a photographer, but as a motion-sensor security system.

Normal Camera: Takes a photo of the whole room every second, even if nothing moved.
Event Camera: Only "blinks" when something changes. If a pixel sees light get brighter or darker, it sends a tiny signal (an "event"). If the room is still, the camera is silent.

This is amazing because:

It reacts in microseconds (super fast).
It works in the dark (it doesn't need a floodlight).
It only records the changes, so it doesn't get overwhelmed by data.

3. The Challenge: The "Noise" in the Signal

Here is the tricky part. When an object vibrates, the event camera sees a chaotic cloud of dots (events) flying around. It's like watching a swarm of bees buzzing around a flower.

Old methods tried to turn these buzzing bees back into a smooth video or a simple line. They often got the speed (frequency) right, but they completely messed up the size of the vibration (amplitude). It was like hearing a song but not knowing if the singer was whispering or screaming.

4. The Magic Trick: Topology and "Shape-Shifting"

This is where the paper's new invention comes in. The researchers used a mathematical tool called Topological Data Analysis (TDA).

The Analogy: The "Connect-the-Dots" Game
Imagine the chaotic swarm of bees (the event data) is a messy pile of string on the floor.

Old methods tried to guess the shape of the string by looking at individual strands.
The New Method uses a "Mapper" algorithm. Imagine you have a giant, flexible net. You drop the net over the pile of string.
1. Covering: You slice the pile into overlapping sections (like cutting a loaf of bread).
2. Clustering: Inside each slice, you group the string that belongs together.
3. The "Centroid": You find the exact center point of each group of string.
4. Connecting: You draw a line connecting these center points.

Suddenly, the messy pile of string transforms into a smooth, clear line that traces the exact path of the vibration. Because they are looking at the shape (topology) of the data rather than just the raw numbers, they can figure out exactly how far the object moved (amplitude) and how fast (frequency).

5. The Results: Hearing Two Voices at Once

The best part? This new "Visual Microphone" can hear multiple things at once.

In their experiment, they put two speakers side-by-side. One played a low note (100 Hz), and the other played a slightly higher note (120 Hz).

A normal microphone would hear a muddy mix of both sounds.
This camera looked at the vibrating cone of the left speaker and the right speaker separately. It drew two distinct lines in the "bee swarm" and reconstructed two separate sounds.

It's like being in a crowded room and being able to focus your eyes on one person's lips to hear exactly what they are saying, while ignoring everyone else.

Why Does This Matter?

This technology is a game-changer because:

It's Passive: It doesn't need to shine a laser or a light on the object. It just watches.
It's Cheap: Event cameras are becoming affordable.
It's Precise: It can measure vibrations that are too small for the human eye to see, helping engineers check if bridges are safe, or doctors check if a heart valve is working, all without touching the object.

In short: They taught a camera to "listen" by watching how light dances on a vibrating object, using a clever math trick to turn a chaotic swarm of dots into a clear, high-fidelity sound wave.

1. Problem Statement

Accurate, non-contact vibration measurement is critical for structural health monitoring, industrial diagnostics, and materials science. While Laser Doppler Vibrometers (LDVs) offer high precision, they are expensive and limited to single-point measurements. High-speed cameras are a more accessible alternative but suffer from trade-offs: achieving high temporal resolution requires short exposure times and intense supplemental lighting, often necessitating reduced regions of interest (ROI) or pixel binning, which degrades spatial resolution.

Event cameras (neuromorphic sensors) offer a solution by recording asynchronous pixel-level brightness changes with microsecond precision, enabling high temporal and spatial resolution without intense lighting. However, existing event-based vibration reconstruction methods face significant limitations:

Active Methods: Rely on laser speckle patterns, which are difficult to apply to multiple simultaneous sources and require complex optical setups.
Passive Methods: Existing passive approaches (e.g., using Gabor filters or converting events to pseudo-frames) can recover principal frequency components but fail to accurately reconstruct vibration amplitude. They often lack a direct geometric link between the event stream and physical displacement.

2. Methodology

The authors propose a passive, event-topology-based visual microphone that directly reconstructs vibration trajectories (amplitude and frequency) from raw event streams without external illumination. The core innovation lies in applying Topological Data Analysis (TDA) to the geometric structure of the event data.

The Reconstruction Pipeline

The process operates on a defined Region of Interest (ROI) centered on the point of highest event density:

Preprocessing:
- Event Filtering: Only "positive" events (brightness increases) are retained to reduce redundancy; negative events are discarded.
- Noise Reduction: A background activity filter (threshold $T = 200 \mu s$ ) removes noise.
Mapper Algorithm Integration:
- Filtering: The 3D spatiotemporal event data ( $x, y, t$ ) is mapped to a 1D space using a filter function. The authors use the vertical position ( $y$ ) as the filter, assuming vertical vibration is the primary mode.
- Covering: The range of the filter function is covered by a set of overlapping intervals.
- Clustering: Within each interval, the HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) algorithm is applied. Unlike DBSCAN, HDBSCAN adapts to varying event densities, making it robust against changes in vibration amplitude and frequency.
Trajectory Reconstruction:
- Instead of generating a graph (standard Mapper output), the method computes the centroid of each cluster.
- These centroids are chronologically connected to form a discrete 3D vibration trajectory.
Projection and Resampling:
- The 3D trajectory is projected onto the $y-t$ plane (discarding horizontal $x$ information) to produce a 1D displacement waveform.
- The signal is resampled to a uniform 16 kHz rate via linear interpolation.

3. Key Contributions

Direct Amplitude Reconstruction: Unlike prior passive methods that struggle with amplitude recovery, this topology-based approach directly maps the geometric structure of event clusters to physical displacement, enabling high-fidelity amplitude reconstruction.
Topology-Aware Framework: The novel integration of the Mapper algorithm with HDBSCAN allows the system to handle complex, non-stationary vibrations where event density fluctuates.
Multi-Source Separation: The method enables the simultaneous recovery of multiple distinct sound sources from a single event stream by identifying and processing separate ROIs for each source.
Passive Operation: The system requires no active illumination (lasers) or external light sources, relying solely on ambient light and the target's natural vibration-induced brightness changes.

4. Experimental Results

The system was validated using an aluminum plate actuated by a mechanical driver and recorded by an event camera (SilkyEvCam HD), with ground truth provided by an LDV.

Test Conditions:
1. Sinusoidal vibration with increasing amplitude.
2. Sinusoidal vibration with decreasing amplitude.
3. Human speech signal ("Mary had a little lamb").
Metrics:
- Normalized Cross-Correlation: Measures waveform shape similarity.
- Median Vector Angle (MVA) Error: Measures spectral similarity (lower is better).
Performance Comparison:
- Vs. Baselines: The proposed method significantly outperformed both a frame-based approach (Davis et al.) and a previous event-based phase method (Niwa et al.).
- Quantitative Gains:
  - Cross-Correlation: Achieved scores of 0.97 (increasing), 0.85 (decreasing), and 0.46 (voice), compared to <0.35 for baselines.
  - MVA Error: Achieved errors between 0.22–0.53 rad, whereas baselines ranged from 1.13–1.54 rad.
Multi-Source Experiment: The system successfully separated and reconstructed two simultaneous tones (100 Hz and 120 Hz) from two different speakers, demonstrating distinct spectral peaks for each source.

5. Significance and Limitations

Significance:
This work advances the frontier of passive vibration sensing by solving the long-standing challenge of amplitude reconstruction in event-based visual microphones. By leveraging topological data analysis, it provides a robust, illumination-free method capable of analyzing complex, multi-source dynamic systems. This has broad implications for non-destructive testing, acoustic monitoring in low-light environments, and structural health monitoring.

Limitations:

Out-of-Plane Motion: The current method relies on 2D image plane displacement. Vibrations occurring strictly along the optical axis (out-of-plane) produce negligible displacement in the 2D projection and cannot be detected.
ROI Sensitivity: Performance degrades if the ROI is too narrow (insufficient events for clustering) or too large (introducing noise). An optimal ROI of approximately 5x25 pixels was identified.

Future Work:
The authors suggest addressing the out-of-plane limitation by incorporating stereo event-based imaging, which would allow for 3D trajectory reconstruction and the measurement of vibrations along the optical axis.