Estimating sub-frame time differences in camera image sequences

This paper presents and validates a cross-spectral technique capable of estimating sub-frame relative time delays between optical intensity signals with better than 50-microsecond accuracy, a method developed for dynamic auroral imaging but applicable to various camera timing calibration and time-varying signal measurement tasks.

The Gist

Imagine you are watching a movie on your phone. The movie isn't actually a continuous stream of light; it's a rapid series of still pictures (frames) shown 60 times every second. To your eyes, it looks smooth, but to a computer, the world is chopped up into tiny slices of time.

Usually, if you want to know exactly when two things happened in a video, you can only be sure they happened within the same "slice" (about 16 milliseconds apart). But what if you need to know if one event happened just a tiny fraction of a second before the other, even though they both appear in the same picture frame?

This is the problem the paper solves. The authors, researchers from the University of Tromsø, have invented a mathematical "super-sense" that lets a camera see time differences much smaller than the time it takes to snap a single photo.

The Problem: The "Flickering" Sky

The researchers were originally thinking about the Aurora Borealis (Northern Lights). Sometimes, the lights in the sky flicker or change shape very quickly. Scientists believe these changes happen because high-speed electrons rain down from space, hitting different layers of the atmosphere at slightly different times.

If you have two cameras watching the sky, or even one camera looking at two different parts of the sky, you might see a "flicker" in one spot and then a "flicker" in another spot a few milliseconds later. Standard cameras are too slow to catch this tiny gap; they just see it as one big blur. The researchers wanted a way to measure that tiny gap without needing expensive, super-fast military-grade cameras.

The Solution: The "Cross-Spectral" Ear

Instead of trying to take a picture faster, the authors used a clever trick based on sound waves and music.

Think of the changing brightness of the aurora (or a flashing light) like a song. Even if the song is playing, it has a rhythm and a beat.

1. The Setup: They built a simple device with two LED lights. One light blinked randomly, and the other light blinked in the exact same pattern, but with a tiny, known delay (like a drummer hitting a snare drum a split-second after the hi-hat).
2. The Recording: They filmed this with a standard smartphone camera.
3. The Magic: They didn't look at the video frame-by-frame. Instead, they took the "song" of the brightness from the first light and the "song" of the second light and compared them mathematically. This is called a cross-spectrum.

The Analogy: Imagine two people clapping their hands. If they clap at the exact same time, their sounds match perfectly. If one person claps a tiny bit later, their sound is slightly out of sync. By listening to the pattern of the claps over a long time, you can calculate exactly how many microseconds one person is lagging behind the other, even if you can't hear the individual claps clearly.

The math works the same way with light. By analyzing the "rhythm" of the light changes across many frames, they could calculate the time difference between two points on the screen with incredible precision.

The Results: Seeing the Invisible

They tested this method and found:

• Extreme Precision: They could measure time differences as small as 50 microseconds (that's 0.00005 seconds). To put that in perspective, a standard video frame lasts about 16,000 microseconds. They are measuring gaps that are 300 times smaller than a single frame.
• The "Rolling Shutter" Effect: They also used this to look at the camera itself. Most smartphone cameras don't take a picture of the whole scene at once; they scan it from top to bottom (like a rolling shutter on a garage door). This means the top of the photo is taken a tiny bit earlier than the bottom. The researchers used their method to map exactly how much time passes as the camera "scans" down the screen, proving they could see the camera's own internal timing quirks.

Why This Matters (According to the Paper)

The paper claims this technique is a game-changer for:

1. Studying the Aurora: It allows scientists to measure the tiny delays in the Northern Lights caused by electrons traveling through the atmosphere, which was previously impossible with standard video.
2. Camera Calibration: It can be used to check if different cameras are perfectly synchronized or to measure the internal timing of a single camera's sensor.

The authors emphasize that this works with cheap, everyday equipment (like a smartphone and a simple Arduino microcontroller) and doesn't require expensive hardware. They successfully proved that by looking at the pattern of light changes rather than just the pictures themselves, we can "hear" time passing in fractions of a millisecond.

Technical Summary

Technical Summary: Estimating Sub-frame Time Differences in Camera Image Sequences

Problem Statement
Optical measurements of dynamic phenomena, such as auroral emissions, often require relative timing accuracy significantly finer than the camera's frame period. Recent electron-transport calculations suggest that prompt auroral emissions can exhibit millisecond-scale time shifts due to finite electron velocities and energy-degradation effects. Resolving these shifts requires synchronization better than 1 ms, and ideally better than 100 µs. Traditional synchronization methods using GPS pulse-per-second signals typically achieve accuracies around 5 ms, which is insufficient for high-speed, multi-wavelength observations of dynamic auroras (e.g., flickering aurora, auroral curls). Furthermore, existing high-speed imaging often relies on single-instrument observations, avoiding the complexity of inter-camera synchronization, but this limits the ability to compare different emission wavelengths simultaneously.

Methodology
The authors propose a cross-spectral technique to estimate relative time delays ($\tau$) between two time-varying optical intensity signals, $I_1(t)$ and $I_2(t)$, where $I_2(t) = \gamma I_1(t - \tau)$. The method is analogous to geodetic very long baseline interferometry used in radio astronomy.

1. Theoretical Basis: The relationship between the time delay and the phase shift in the frequency domain is linear ($\phi_{12}(\omega) = \omega\tau$). By computing the cross-spectrum of two signals, the phase of the mean cross-spectrum provides information about the relative time delay.
2. Estimation Process:
   • Discrete Fourier Transforms (DFT) are applied to pixel-intensity time series.
   • The cross-spectral phase is calculated across multiple frequency components.
   • A linear least-squares model (maximum-likelihood estimate) is fitted to the phase data to solve for $\tau$.
   • Phase unwrapping is applied if the delay is large enough to cause phase wrapping, provided measurement errors are small ($\sigma \ll \pi$).
3. Validation Setup: To validate the technique, the authors constructed a calibration device using an Arduino UNO microcontroller to drive two LEDs. The LEDs emit pseudorandomly pulsed light with a user-defined relative time shift. The signals are diffused to create uniform illuminated regions.
4. Data Acquisition: Recordings were made using a Huawei P30 Pro smartphone camera (60 fps, 2336×1080 resolution) in a dark room. The device generated signals with known delays, which were recorded and analyzed.

Key Results

• Accuracy: For tested recordings, the method estimates relative delays between image-sensor regions with an accuracy better than 50 µs. In specific tests with 104 image frames (approx. 167 s duration), the absolute accuracy was approximately 30 µs.
• Error Dependence: Measurement uncertainty decreases as the recording duration increases. For a 1.05 ms delay, the standard deviation of the estimate dropped from 279 µs (5 s duration) to 26 µs (167 s duration).
• Sensor Characterization: The technique successfully characterized the rolling-shutter readout of the smartphone camera. By analyzing a 200×200 pixel region, the authors observed a linear increase in time delay across image rows. The average time-shift change between adjacent rows was 6.58 µs, consistent with a plausible row-readout time. The standard deviation of relative time-delay errors between adjacent pixels was found to be less than 10 µs.
• Robustness: The method was tested with different time delays (1.05 ms, 2.05 ms, 10.05 ms) and different cameras (iPhone X at 30 fps), yielding consistent results, though the iPhone X produced slightly larger errors due to a noisier sensor and lower frame rate.

Significance and Claims
The paper claims that this technique provides a feasible, low-cost solution for measuring sub-frame time shifts in digital video sequences.

• Auroral Science: The demonstrated accuracy (<50 µs) is sufficient to study electron-transport effects in dynamic aurora, where relative delays between prompt emissions from different altitudes are expected to be several milliseconds. The method allows for the measurement of these delays within 100 µs for events lasting less than 20 seconds, provided the emission has sufficient temporal bandwidth (approx. 10 Hz).
• Camera Calibration: The technique can characterize sub-frame timing differences across an image sensor (e.g., rolling-shutter effects) and can be used for inter-camera and inter-sensor time calibration when cameras are synchronized to a shared clock.
• General Applicability: While developed for auroral imaging, the authors note the technique has broader applications in any field requiring the measurement of time-varying optical signals or camera timing calibration.

The authors maintain a modest tone regarding future implications, noting that application to dynamic aurora and time-dependent electron transport is a topic for future work, and that the specific accuracy values depend on the camera, compression algorithm, and setup used. They do not claim the method eliminates the need for shared clocks in multi-camera setups but rather provides a way to characterize and correct for sub-frame timing offsets when such synchronization exists.