💻 computer science

Seeing Fast and Slow: Learning the Flow of Time in Videos

This paper introduces self-supervised models that learn to detect and control video playback speed, leveraging these capabilities to curate a large slow-motion dataset and enable advanced applications like speed-conditioned video generation and temporal super-resolution.

Original authors: Yen-Siang Wu, Rundong Luo, Jingsen Zhu, Tao Tu, Ali Farhadi, Matthew Wallingford, Yu-Chiang Frank Wang, Steve Marschner, Wei-Chiu Ma

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yen-Siang Wu, Rundong Luo, Jingsen Zhu, Tao Tu, Ali Farhadi, Matthew Wallingford, Yu-Chiang Frank Wang, Steve Marschner, Wei-Chiu Ma

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a movie. Sometimes, the action feels frantic and fast-forwarded, like a squirrel on espresso. Other times, it feels like a dream, where a falling leaf takes ten seconds to hit the ground. Humans are naturally good at sensing this "flow of time." If a video is sped up, we know it feels "wrong" because the water ripples too fast or a person falls like a stone.

But for computers, time is a mystery. Most AI models have only ever seen videos played at a standard speed (like 30 frames per second). They don't know what "slow motion" actually is; they just see a sequence of pictures. If you ask them to make a video in slow motion, they often just play the same video slower, which looks choppy and unnatural, or they ignore your instructions entirely.

This paper, "Seeing Fast and Slow," is like teaching a computer to finally understand the concept of time. The researchers from Cornell, NTU, and UW built a system that can not only tell if a video is fast or slow but also create new videos at any speed you want, from normal speed to extreme slow motion.

Here is how they did it, explained through simple analogies:

1. The "Audio Detective" (Learning to Spot Speed Changes)

The Problem: How do you teach a computer to spot when a video speeds up or slows down without having a human label every single second?
The Solution: The researchers realized that sound and motion are best friends.

The Analogy: Think of a record player. If you speed up the record, the music sounds higher-pitched and chipmunk-like. If you slow it down, the voice sounds deep and demonic.
The Trick: The computer listens to the video's audio. When the pitch suddenly jumps up, the computer knows, "Ah! The video just sped up!" When the pitch drops, it knows, "It just slowed down!"
The Result: By using this audio clue, the computer taught itself to find exactly when the speed changed in thousands of videos, creating a massive library of "speed-change" examples without needing human help.

2. The "Time Traveler's Math" (Learning to Guess the Speed)

The Problem: Once the computer knows when the speed changed, how does it know how much it changed? Is it 2x faster? 10x slower?
The Solution: They used a clever math trick called equivariance.

The Analogy: Imagine you have a video of a runner. If you play it at 2x speed, it looks like the runner is running twice as fast. If you then take that 2x video and play it at 2x speed again, it should look like the runner is going 4x speed. The relationship is predictable.
The Trick: The computer takes a video, speeds it up artificially, and asks itself, "If I sped this up by 2x, my prediction should also go up by 2x." It practices this over and over, learning to estimate the exact speed just by looking at how things move.
The Result: The computer became so good at this that it can guess the speed of a video almost as accurately as a human expert.

3. The "Time Machine" (Building the SloMo-44K Dataset)

The Problem: To teach a computer to create slow motion, you need to show it real slow motion. But real slow-motion videos (filmed with expensive high-speed cameras) are rare and messy on the internet.
The Solution: They used their new "Speed Detective" and "Time Traveler" tools to scour the internet.

The Analogy: Imagine a librarian who can instantly find every single book in a library that was written in a specific dialect, even if the books aren't labeled.
The Trick: They scanned millions of videos from YouTube and Vimeo. Their AI filtered out the normal stuff and kept only the true slow-motion clips. They ended up with SloMo-44K, a massive dataset of 44,000 slow-motion clips. This is the largest collection of its kind, containing 18 million frames of high-speed action.

4. The "Time Sculptor" (Creating and Fixing Videos)

With this new knowledge and the massive dataset, they built two superpowers:

Speed-Conditioned Generation:
- What it does: You give the AI a picture and a text prompt (e.g., "A bird landing"), and you tell it, "Make this happen in extreme slow motion."
- The Magic: Unlike old models that just stretch the video, this model understands physics. It knows that in slow motion, a bird's wings move slowly, and water droplets hang in the air. It generates brand new frames that look like they were filmed with a $10,000 high-speed camera.
Temporal Super-Resolution (The "De-Blur" Magic):
- What it does: You give it a blurry, low-quality video (like a shaky phone recording of a car crash) and ask it to turn it into a crisp, high-speed slow-motion video.
- The Magic: Standard AI just guesses what the missing frames look like, often making them blurry. This model, trained on real high-speed data, knows exactly how motion blurs and how to "un-blur" it, revealing fine details that were previously invisible.

Why Does This Matter?

Think of this as giving computers a sense of time.

For Forensics: It could help police figure out if a security camera video has been tampered with or sped up to hide a crime.
For Creativity: Filmmakers could generate slow-motion shots of explosions or sports plays without needing expensive cameras.
For AI Intelligence: It helps AI understand the real world better. If an AI understands how fast a ball falls or how water splashes, it can navigate the physical world much more safely and effectively.

In short, the researchers taught computers to stop just "watching" videos and start truly feeling the passage of time.

1. Problem Statement

Current computer vision models, including advanced Video Large Language Models (VLMs) and generative video models, lack a fundamental understanding of temporal speed. They struggle to:

Perceive Time: Determine if a video is sped up or slowed down, or estimate the specific playback speed.
Control Time: Generate videos at user-specified playback speeds (e.g., extreme slow motion vs. normal speed) or perform temporal super-resolution (converting low-FPS, blurry videos into high-FPS, clear sequences).
Data Scarcity: Existing datasets are dominated by standard frame rates (24–60 fps). High-frame-rate (slow-motion) data is scarce, noisy, and lacks reliable metadata (e.g., exact frame rates or playback speeds), making supervised learning difficult.

The core limitation identified is that models are trained on a single "cadence" of time, preventing them from learning the continuum of temporal speeds required to reason about real-world physics.

2. Methodology

The authors propose a three-stage framework: Self-Supervised Perception, Dataset Curation, and Temporal Manipulation.

A. Self-Supervised Perception Models

To overcome the lack of labeled data, the authors develop two models using intrinsic video cues:

Speed Change Detection (Audio-Visual Cross-Modal):
- Insight: Based on the principle of time-frequency scaling, changing a video's playback speed shifts its audio pitch (speeding up raises pitch; slowing down lowers it).
- Method: The model uses audio pitch shifts to automatically generate labels for moments where playback speed changes. A visual detector (based on VideoMAEv2) is trained to locate these transitions using only visual input during inference.
Playback Speed Estimation (Temporal Equivariance):
- Insight: Speed estimation should be equivariant to temporal resampling. If a video is accelerated by a factor $k$ , the predicted speed should scale by $k$ .
- Method: The model is trained with a self-supervised loss function that enforces the relationship: $f_\theta(k \cdot V) \approx k \cdot f_\theta(V)$ .
- Refinement: To handle extreme slow-motion where motion cues are subtle, an Iterative Prediction strategy is used. The model predicts a speed, accelerates the video to bring it closer to normal speed, and re-estimates, repeating this process to refine accuracy.

B. Dataset Curation: SloMo-44K

Using the trained perception models, the authors automatically curated SloMo-44K, the largest generic slow-motion dataset to date.

Scale: 44,632 clips, 18 million frames, spanning diverse scenes and motion patterns.
Quality Control: Videos are sourced from the web, filtered for CGI/screen recordings, segmented into homogeneous speed clips using the speed-change detector, and annotated with predicted playback speeds.
Significance: Unlike previous datasets (e.g., Adobe240fps, SportsSloMo) which are small or domain-specific, SloMo-44K covers a wide range of temporal scales (up to 10,000+ FPS capture) and general scenarios.

C. Manipulating the Flow of Time

The curated dataset is used to train models for two generation tasks:

Speed-Conditioned Video Generation:
- Architecture: Built upon Wan2.1-I2V.
- Mechanism: Playback speed is discretized into logarithmic buckets. These are encoded via sinusoidal positional embeddings and injected into the model's timestep embedding and latent features via MLPs.
- Goal: Generate physical dynamics (e.g., fluid, shattering) at specific speeds (0.01× to 1.0×), ensuring the motion physics match the requested speed rather than just slowing down a normal video.
Extreme Temporal Super-Resolution:
- Architecture: Built upon Wan2.1-VACE.
- Challenge: Real-world low-FPS videos suffer from motion blur.
- Method: The model is trained on synthetic low-FPS, motion-blurred inputs (generated by averaging frames from SloMo-44K) to learn joint motion deblurring and frame interpolation.

3. Key Contributions

New Visual Concept: Establishes "time" as a learnable and manipulable visual dimension in computer vision.
Self-Supervised Learning: Introduces novel self-supervised objectives (audio-pitch for change detection, temporal equivariance for speed estimation) that eliminate the need for manual speed labels.
SloMo-44K Dataset: Releases the largest and most diverse slow-motion dataset, enabling training on a continuum of temporal speeds.
State-of-the-Art Performance: Demonstrates that models trained on this data can generate controllable slow-motion and perform super-resolution on blurry inputs, outperforming existing baselines.

4. Experimental Results

Perception Tasks

Speed Change Detection: Achieved 92.4% accuracy, significantly outperforming Gemini 2.5 (59.5%) and optical flow baselines (80.4%).
Playback Speed Estimation: Achieved a Pearson correlation ( $\rho$ ) of 0.735 and Spearman rank correlation ( $r_s$ ) of 0.706, narrowing the gap between machine and human expert performance (Human $\rho \approx 0.88$ ).

Generation Tasks

Speed-Conditioned Generation:
- The model successfully generated videos where optical flow magnitude correlated strongly with the requested speed (unlike baselines which failed to vary motion intensity).
- Achieved lower FID (68.4) and FVD (1114.1) compared to Wan2.1 and ATI baselines, indicating higher quality and better speed control.
Temporal Super-Resolution:
- Clear Inputs: Outperformed FILM, LDMVFI, and Wan2.1 on standard metrics (FloLPIPS, FVD).
- Blurred Inputs: Significantly outperformed all baselines in reconstructing sharp frames from motion-blurred inputs (Human preference win rate: 80.3%).
- Real-World Inputs: In a user study on real low-FPS footage, the model achieved over 90% preference for realism and smoothness.

5. Significance and Impact

Video Forensics: The ability to detect speed changes and estimate playback rates has direct applications in identifying deepfakes or manipulated media.
World Models: By learning how events unfold at different speeds, models gain a richer understanding of physical dynamics, moving beyond static frame prediction to dynamic temporal reasoning.
Creative Control: Enables precise control over video generation, allowing creators to synthesize specific physical phenomena (e.g., bullet-time effects, fluid dynamics) at arbitrary speeds without relying on high-speed cameras.
Foundation for Future Research: The SloMo-44K dataset and the self-supervised methodologies provide a new foundation for research into temporal reasoning, potentially leading to more robust video understanding and generation systems.