🤖 AI

PAS: A Training-Free Stabilizer for Temporal Encoding in Video LLMs

This paper introduces Phase Aggregated Smoothing (PAS), a training-free method that stabilizes temporal encoding in Video LLMs by applying opposed phase offsets across attention heads to smooth the frequency-induced ripples of Rotary Position Embeddings, thereby enhancing robustness to frame timing shifts with negligible computational overhead.

Original authors: Bowen Sun, Yujun Cai, Ming-Hsuan Yang, Hang Wu, Yiwei Wang

Published 2026-03-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Bowen Sun, Yujun Cai, Ming-Hsuan Yang, Hang Wu, Yiwei Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Wobbly Flashlight" Effect

Imagine you are trying to watch a movie, but the projector is broken. Instead of showing a smooth, continuous film, it flashes the frames on and off in a weird, rhythmic pattern.

The Good Frames: Sometimes the light hits the screen perfectly, and you see the action clearly.
The Bad Frames: Other times, the light dips into a "shadow" (a low-gain trough). Even if a crucial moment happens (like a bird taking flight), the projector dims it so much that your brain (the AI) thinks, "Oh, that frame isn't important," and ignores it.

This is what happens to current Video LLMs (AI models that understand video). They use a mathematical tool called RoPE (Rotary Position Embedding) to understand when things happen in a video. The authors discovered that this tool creates a "ripple effect" in time.

Because of these ripples, if you shift the video by just a tiny fraction of a second (or change the frame rate slightly), the AI might suddenly decide that a key frame is invisible. It's like trying to catch a ball while the ground beneath you is shaking; sometimes you catch it, sometimes you miss, not because of the ball, but because of the shaking ground.

The Solution: PAS (Phase Aggregated Smoothing)

The authors created a fix called PAS. Think of it as a "noise-canceling headphone" for the AI's sense of time.

Here is how it works, using a Choir Analogy:

Imagine the AI has many "heads" (think of them as different singers in a choir), and they are all trying to sing the same song (analyze the video).

The Old Way: Every singer starts exactly at the same beat. If the conductor (the video timeline) stutters, the whole choir stumbles together.
The PAS Way: The conductor asks the singers to start at slightly different times.
- Singer A starts at exactly 0:00.
- Singer B starts at 0:01.
- Singer C starts at 0:02.

Now, when the "ripple" (the glitchy timing) hits, it might make Singer A stumble. But because Singer B and C started a split second later, they are hitting a different part of the ripple. When the AI combines (aggregates) all their voices, the stumbles cancel each other out. The result is a smooth, steady song.

Why is this special?

It's "Training-Free": Usually, to fix an AI, you have to feed it thousands of hours of video and re-teach it everything (like going back to school). PAS doesn't do that. It's a plug-and-play upgrade. You just install it, and it works immediately. It's like putting a stabilizer on a shaky camera without having to rebuild the camera.
It's Cheap: It adds almost no extra work for the computer. It's like adding a tiny filter to a lens; the picture gets better, but the camera doesn't get heavier or slower.
It Doesn't Change the "Truth": The authors proved mathematically that while the singers (heads) are starting at different times, they are all still hearing the exact same notes. They aren't changing what the AI sees, just how it averages the timing to make it more stable.

The Results

The team tested this on nine different video challenges (like recognizing actions, understanding long stories, or spotting objects).

The Result: The AI became much more consistent. It stopped missing important moments just because the video was sampled slightly differently.
The Catch: The fix works best when the video is "sparse" (fewer frames per second). If the video is already super smooth and high-quality, the fix helps less, because there was less shaking to begin with.

Summary in One Sentence

PAS is a clever, free software trick that stabilizes video AI by having different parts of the model "look" at the video at slightly different times and then averaging their answers, effectively smoothing out the jittery math that usually causes the AI to miss important moments.

1. Problem Statement: Temporal Instability in Video LLMs

Video Large Language Models (Video LLMs) often suffer from temporal inconsistency, where minor shifts in frame timing or sampling offsets cause significant fluctuations in model attention and downstream accuracy.

Root Cause: The authors trace this instability to the extension of Rotary Position Embeddings (RoPE) to video via Multimodal RoPE (M-RoPE). M-RoPE assigns a finite set of temporal frequencies to encode time.
The Mechanism of Failure:
- The authors analyze M-RoPE through a Fourier perspective. They demonstrate that the attention logit for a relative time lag $\Delta t$ can be approximated as the raw content dot product multiplied by a time-domain kernel $m(\Delta t)$ , which is the Inverse Fourier Transform (IFT) of the RoPE frequency spectrum.
- Because the spectrum consists of discrete frequency lines, the resulting time-domain kernel $m(\Delta t)$ is intrinsically rippled (oscillatory) at the frame scale.
- Consequence: Small changes in frame timing (lag) cause the model to sample different points on this rippled waveform. This leads to "gain swings" where relevant frames fall into low-gain troughs (suppressing attention) or high-gain peaks, making the model highly sensitive to sampling offsets and frame rates.

2. Methodology: Phase Aggregated Smoothing (PAS)

The authors propose Phase Aggregated Smoothing (PAS), a training-free, inference-time plug-in designed to smooth the temporal modulation kernel without altering the underlying positional encoding structure or requiring retraining.

Core Concept

PAS leverages the property that a time shift in the time domain corresponds to a phase rotation in the frequency domain, leaving the spectrum magnitude unchanged.

Phase Offsets: The method assigns small, opposed temporal phase offsets ( $\delta_h$ ) to different attention heads. Specifically, it applies these offsets only to the Query (Q) stream.
Head Aggregation: Each head observes the same merged-frame tokens but samples the temporal modulation kernel $m(\Delta t)$ at slightly different temporal offsets.
Smoothing Effect: When the model aggregates the outputs of these heads, it effectively performs a controlled moving average of the rippled kernel. This attenuates the high-frequency ripples of $m(\Delta t)$ while preserving the low-frequency trend.

Implementation Details

Input: Tokenized video sequences.
Operation: For each attention head $h$ , apply a phase operator $\Gamma_{\delta_h}$ to the query vector $Q_h$ .
Constraints:
- Offsets are applied only to video tokens (not text).
- Offsets are small (e.g., 0.5 bin units) to ensure the temporal order of frames is preserved across heads.
- The method is compatible with Multi-Head Attention (MHA) and Grouped Query Attention (GQA).
Computational Cost: Negligible. It adds only a linear transform on the temporal half-dimensions of the query, resulting in an overhead ratio of $\approx p_t S_v / S^2$ , which is statistically indistinguishable from the baseline in throughput tests.

3. Theoretical Foundations

The paper provides three key theorems to justify the approach:

Theorem 1 (Phase Modulation Approximation): Formalizes that the RoPE-rotated logit is the content dot product scaled by the real part of the IFT kernel $m(\Delta t)$ .
Theorem 2 (Smooth IFT $\Rightarrow$ Phase-Stable Attention): Proves that if the IFT kernel is smoother (lower Lipschitz constant), the attention logit changes linearly with small time perturbations. Smoothing the kernel reduces sensitivity to timing jitter.
Theorem 3 (Multi-Phase Averaging): Demonstrates that aggregating heads with small, opposed phase shifts mathematically reduces the local temporal variation (ripples) of the effective modulation kernel.
Theorem 4 (Spectral Invariance): Proves that under Nyquist-valid sampling, applying a time shift to a head only rotates the phase of the recovered spectrum, leaving the magnitude (the encoded information) invariant. Thus, PAS changes how the modulation is sampled, not what is encoded.

4. Experimental Results

The authors evaluated PAS on nine benchmarks covering action recognition and general video understanding, using a matched token budget (no increase in computational load).

Datasets: 20BN-Jester, Something-Something V2, Kinetics-700, Breakfast Actions, MVBench, TempCompass, PerceptionTest, EgoSchema, and MMBench-Video.
Baselines: Compared against the original backbone (Qwen2.5-VL-7B) and other training-free methods like SlowFast-LLaVA and TS-LLaVA.
Key Findings:
- Consistent Gains: PAS consistently improved performance across all benchmarks. It achieved the best single-model results on 20BN-Jester and Kinetics-700.
- Stacking: PAS can be stacked with other methods (e.g., SlowFast-LLaVA + PAS), yielding further improvements.
- Parameter Robustness: The method is robust to hyperparameter choices. A simple setting of $K=2$ groups with an offset of $\Delta \approx 0.5$ bin units works well across diverse datasets.
- Sampling Rate Sensitivity:
  - Sparse Sampling: PAS provides the largest gains when sampling is sparse (low FPS), where the "ripples" in the kernel are most pronounced.
  - Dense Sampling: As the sampling ratio increases (approaching Nyquist), the baseline naturally smooths out, and the relative benefit of PAS diminishes, aligning with the theoretical predictions.
- Sub-Nyquist: On undersampled datasets (e.g., Breakfast Actions), gains are smaller but still present, though limited by aliasing effects.

5. Significance and Contributions

Diagnosis: The paper provides a rigorous Fourier-based diagnosis of why standard RoPE fails in continuous-time video contexts, identifying the "rippled kernel" as the source of instability.
Training-Free Solution: PAS offers a plug-and-play upgrade that requires no retraining, no parameter updates, and negligible computational overhead.
Robustness: It significantly enhances the robustness of Video LLMs to frame rate variations and sampling offsets, a critical requirement for real-world deployment where input video quality varies.
Theoretical Guarantee: The work bridges the gap between signal processing theory (Fourier analysis, Nyquist sampling) and deep learning architecture design, proving that smoothing the modulation kernel stabilizes attention without losing semantic content.

In summary, PAS transforms the temporal encoding in Video LLMs from a fragile, phase-sensitive mechanism into a robust, smoothed process, enabling more reliable video understanding under diverse sampling conditions.