💻 computer science

TETO: Tracking Events with Teacher Observation for Motion Estimation and Frame Interpolation

The paper proposes TETO, a teacher-student framework that achieves state-of-the-art event-based motion estimation and frame interpolation using only ~25 minutes of unannotated real-world data by distilling knowledge from a pretrained RGB tracker and leveraging the resulting motion priors to condition a video diffusion transformer.

Original authors: Jini Yang, Eunbeen Hong, Soowon Son, Hyunkoo Lee, Sunghwan Hong, Sunok Kim, Seungryong Kim

Published 2026-03-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jini Yang, Eunbeen Hong, Soowon Son, Hyunkoo Lee, Sunghwan Hong, Sunok Kim, Seungryong Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Blind Spot" of Regular Cameras

Imagine you are watching a movie, but the projector is broken. It skips frames. You see a ball at position A, then suddenly at position B, but you missed the 100 frames in between where the ball actually flew through the air.

Regular Cameras (RGB): These are like the broken projector. They take a picture, wait a fraction of a second, take another, and wait again. If something moves too fast (like a hummingbird's wings or a speeding car), the camera misses the motion in between. It's a "blind spot."
Event Cameras: These are like a super-sensitive security guard. Instead of taking full photos, they only shout out when something changes. "Pixel 5,5 changed brightness!" "Pixel 5,6 changed!" They shout thousands of times per second with microsecond precision. They never miss a movement, even in the dark or at super-high speeds.

The Catch: While Event Cameras are amazing at seeing motion, we don't have a good way to teach computers how to understand them. Usually, we have to train AI on massive amounts of fake (synthetic) data because real-world event data is hard to label. But fake data is like a cartoon; it doesn't look or feel exactly like the real world, so the AI gets confused when it tries to work in real life.

The Solution: TETO (The Teacher-Student Trick)

The authors created TETO (Tracking Events with Teacher Observation). Think of it as a master-apprentice relationship.

1. The Teacher (The Expert)

The "Teacher" is a super-smart AI that has already learned how to track motion using regular video (RGB). It's an expert at following objects in normal movies. It knows exactly where a ball is and where it's going.

2. The Student (The Event Camera AI)

The "Student" is the new AI we want to train. It only sees the "shouts" from the Event Camera (the raw data), which looks like static noise to a human. It doesn't know what an object looks like; it only knows where things are moving.

3. The Magic Lesson (Knowledge Distillation)

Here is the genius part:

The researchers took a tiny amount of real-world video (about 25 minutes).
They fed the regular video to the Teacher. The Teacher said, "Okay, the ball is here, then here, then here."
They fed the event camera data (from the same 25 minutes) to the Student.
The Teacher whispered the answers to the Student: "When you see these specific flashes of light, the ball is actually moving this way."

The Analogy: Imagine a master chef (Teacher) tasting a complex dish and telling an apprentice (Student) exactly how to season it, even though the apprentice is blindfolded and can only feel the texture of the ingredients. The apprentice learns the motion of the cooking, not just the look of the food.

Because the Teacher is so good, the Student learns incredibly fast. They only needed 25 minutes of real data to learn what other methods needed hours of fake data to learn.

The Secret Sauce: "Motion-Aware" Learning

One problem with the 25 minutes of video is that most of the time, the camera itself is moving (like a car driving down the street). This is called "ego-motion." It's like trying to learn how a fish swims while you are on a moving boat; the water looks like it's moving, but it's actually the boat.

TETO has a special filter:

It looks at the Teacher's data and says, "Okay, the whole background is moving because the camera is moving. Ignore that."
It focuses only on the objects that are moving independently (like a ball or a person).
It forces the Student to pay extra attention to these moving objects. This prevents the AI from getting lazy and just guessing that everything is moving because the camera is moving.

The Payoff: Super-Fast Video Interpolation

Once the Student learns how to track motion perfectly, the researchers use it for a cool trick: Frame Interpolation.

The Problem: You have two photos of a fast-moving car. You want to create a smooth video between them, but you don't have the frames in the middle.
The Old Way: AI tries to guess what the middle looks like, often resulting in blurry smears or "ghosts."
The TETO Way: Because TETO knows the exact path the car took (thanks to the Event Camera data), it can tell the video generator: "The car was here at 1:00 and there at 1:01. Draw the car smoothly connecting those two dots."

It's like having a GPS that knows the exact route a car took, so you can draw the road perfectly between two points, even if you missed the drive.

Summary of Results

Less Data, More Power: TETO trained on 25 minutes of real data and beat methods trained on 5 hours of fake data.
Better Tracking: It tracks moving objects better than anyone else on difficult datasets (EVIMO2).
Smoother Videos: It creates much sharper, less blurry videos when filling in the gaps between frames (Frame Interpolation).

The Bottom Line

TETO is a smart way to teach a computer to "see" motion using a tiny bit of real-world data and a lot of help from an expert AI. It bridges the gap between the "blurry" world of regular cameras and the "hyper-fast" world of event cameras, allowing us to create smoother, more realistic videos of fast-moving things.

1. Problem Statement

Event cameras offer microsecond temporal resolution and capture per-pixel brightness changes, providing continuous motion information that is lost between discrete RGB frames. This makes them ideal for high-speed motion estimation and video frame interpolation (VFI). However, existing event-based motion estimators face two critical challenges:

Sim-to-Real Gap: Current state-of-the-art methods (e.g., ETAP, MATE) rely heavily on large-scale synthetic data (hours of rendered sequences). Synthetic events often fail to capture the complex, continuous temporal dynamics of real-world events, leading to artifacts and poor generalization.
Data Scarcity: Real-world event-RGB paired datasets are scarce and lack motion annotations (ground truth trajectories or optical flow), making supervised training difficult.

The core question addressed is: How can we achieve accurate event-based motion estimation using only a small amount of unannotated real-world data?

2. Methodology: TETO Framework

The authors propose TETO (Tracking Events with Teacher Observation), a teacher-student framework that learns event motion estimation from only ~25 minutes of unannotated real-world recordings.

A. Teacher-Student Distillation

Teacher: A pretrained RGB tracker (AllTracker) that generates pseudo-labels (point trajectories and dense optical flow) on paired RGB frames.
Student: An event motion estimator adapted from AllTracker. It takes event stacks as input and learns to predict the same outputs as the teacher.
Mechanism: Instead of training from scratch, the student is fine-tuned via knowledge distillation. The student learns to map event temporal structures to motion correspondences, inheriting the teacher's matching capabilities while leveraging the unique continuous motion cues of events.

B. Motion-Aware Data Curation & Query Sampling

To maximize learning from limited data (~25 mins), the authors introduce specific strategies to handle the imbalance between dominant ego-motion (camera movement) and object motion:

Ego-Motion Decomposition: The teacher's optical flow is decomposed using RANSAC to estimate a global affine model. The residual flow is used to generate a binary mask ( $M_{obj}$ ) identifying independently moving objects.
Query Oversampling: During training, 90% of query points are sampled from $M_{obj}$ (dynamic regions) rather than uniformly across the frame. This prevents the model from overfitting to global background motion and forces it to learn complex object interactions.
Strict Curation: Only sequences with precise sensor alignment, high resolution, and diverse motion dynamics (e.g., EHPT-XC and ERF-X170FPS datasets) are selected.

C. Architecture Adaptation

Concentration Network: Since event stacks are multi-channel (temporal bins) and the RGB encoder expects 3 channels, a lightweight U-Net (Concentration Network) aggregates multi-scale event information into a 3-channel representation. This allows the use of the pretrained ConvNeXt encoder without architectural modification.
Joint Prediction: The model jointly predicts sparse point trajectories (with visibility) and dense optical flow in a single forward pass.

D. Application: Motion-Conditioned Video Frame Interpolation (VFI)

The estimated motion signals are used to condition a pretrained Video Diffusion Transformer (DiT) for frame interpolation:

Latent Warping: Dense optical flow from TETO warps boundary frame latents for coarse spatial alignment.
Trajectory Guidance: Point trajectories supervise the DiT's self-attention maps, enforcing geometric consistency and fine-grained correspondence during denoising.
Event Motion Mask ( $M_{event}$ ): A mask derived directly from raw event accumulations (without learned components) identifies dynamic regions, directing the generation capacity toward moving objects.

3. Key Contributions

Data-Efficient Learning: Demonstrated that high-quality event motion estimation can be achieved with ~25 minutes of real data, a reduction of orders of magnitude compared to synthetic-data-dependent methods (which use 1.5–5 hours).
Teacher-Student Framework: Introduced a novel distillation pipeline that transfers correspondence matching abilities from RGB trackers to the event domain, effectively bridging the modality gap without manual annotations.
Motion-Aware Sampling: Proposed a strategy to disentangle object motion from ego-motion, ensuring robust learning of dynamic object tracking in limited data regimes.
Unified Motion-to-Generation Pipeline: Showed that accurate event-based motion estimation serves as a powerful prior for video diffusion models, significantly improving frame interpolation quality in dynamic scenes.

4. Experimental Results

The method was evaluated on multiple real-world benchmarks:

Point Tracking (EVIMO2): TETO achieved State-of-the-Art (SOTA) performance (AJ: 67.9, $\delta_x^{avg}$ : 81.4, OA: 92.2), outperforming ETAP which was trained on 5 hours of synthetic data.
Optical Flow (DSEC):
- Zero-shot: Outperformed several unsupervised/self-supervised methods trained on in-domain data, despite never seeing driving data during training.
- In-domain: Surpassed all compared methods in End-Point Error (EPE) and Angular Error (AE) after fine-tuning with pseudo-labels.
Video Frame Interpolation (BS-ERGB & HQ-EVFI):
- Achieved the best perceptual quality (lowest FID and LPIPS) on highly dynamic scenes.
- Qualitative results showed sharper boundaries and fewer ghosting artifacts compared to warping-based (TimeLens) and diffusion-based (RE-VDM) baselines.
Generalization: TETO succeeded in extreme conditions (nighttime, high speed, water droplets) where the RGB teacher failed, proving the student learned event-specific temporal cues rather than just mimicking RGB appearance.

5. Significance

Paradigm Shift: TETO challenges the prevailing notion that event-based vision requires massive synthetic datasets. It proves that data quality and curation (real-world motion diversity) can outweigh data scale (synthetic volume).
Bridging Modalities: It provides a robust pathway to transfer knowledge from mature RGB vision models to the emerging event camera domain.
Practical Impact: By enabling high-quality motion estimation and frame interpolation with minimal data, TETO makes event-based vision more accessible for real-world applications in robotics, autonomous driving, and high-speed video synthesis, particularly in scenarios where ground truth is unavailable.