💻 computer science

SemanticMoments: Training-Free Motion Similarity via Third Moment Features

To address the failure of existing models to disentangle motion from appearance, the authors propose **SemanticMoments**, a training-free method that achieves superior motion-based video retrieval by computing higher-order temporal statistics over pre-trained semantic features.

Original authors: Saar Huberman, Kfir Goldberg, Or Patashnik, Sagie Benaim, Ron Mokady

Published 2026-02-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Saar Huberman, Kfir Goldberg, Or Patashnik, Sagie Benaim, Ron Mokady

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a video of a person drinking coffee. If I asked you to find another video that is "similar," most AI models today would show you another person in a kitchen, perhaps wearing similar clothes, or sitting at a similar table. They are looking at the "scenery" (the coffee cup, the kitchen, the person's shirt).

But what if I asked you to find a video that has the same motion? You wouldn't care if the person was a man or a woman, or if they were in a cafe or a park. You would look for the specific rhythm and pattern of the arm lifting the cup to the mouth.

This paper, "SemanticMoments," is about teaching AI to stop being "distracted by the scenery" and start focusing on the "dance" of the action.

The Problem: The "Photographic Memory" Bias

Current AI models are like people with incredible photographic memories but very little sense of rhythm.

Think of it like this: If you show an AI a video of a person playing a cello, the AI sees the "cello" and the "musician" and says, "Aha! Music!" It doesn't actually need to see the bow moving to know what's happening; it just recognizes the objects. This is called Appearance Bias. Because of this, if you ask the AI to find "the motion of playing a cello," it might just give you a thousand pictures of different cellos, even if the people in them are sitting perfectly still.

On the other hand, older methods tried to look only at "pixels moving" (like watching a grainy black-and-white security camera). These models see the movement, but they have no idea what is moving. They can't tell the difference between a person waving their hand and a tree swaying in the wind.

The Solution: SemanticMoments (The "Rhythm Tracker")

The researchers created a new way to describe video called SemanticMoments. Instead of just looking at a single average of the video, they look at the "Statistical Moments" of the features.

To understand this, let's use a Metaphor: The Rollercoaster.

Imagine you are describing a rollercoaster ride to a friend:

The First Moment (The Average): This is like saying, "The ride stays at a medium height." It tells you the general location, but nothing about the excitement. (This is what most AI does now).
The Second Moment (The Variance): This is like saying, "The ride goes up and down a lot!" It captures the energy and the intensity of the movement.
The Third Moment (The Skewness): This is like saying, "The ride has sudden, sharp drops rather than smooth hills." It captures the shape and the direction of the movement.

By combining these three "moments," the researchers create a mathematical "fingerprint" of the motion. It doesn't care if the rollercoaster is red or blue, or if it's in Florida or France; it only cares about the pattern of the ups and downs.

How they proved it works

The researchers did two things to test their "Rhythm Tracker":

The "SimMotion-Synthetic" Test: They used AI to create "trick" videos. They made videos where the motion was identical (e.g., a person walking), but they changed everything else—the art style, the clothes, the camera angle, and even the person. They found that while other AIs got confused by the new clothes or styles, SemanticMoments correctly identified that the "walk" was the same.
The "SimMotion-Real" Test: They tested it on real-world videos. Even when the videos were messy and uncoordinated, SemanticMoments was much better at finding the "soul" of the movement than the existing industry standards.

Why does this matter?

This isn't just about finding videos. This technology is a building block for:

Better AI Video Creators: Helping tools like Sora or Runway understand exactly how a specific movement should look.
Robotics: Helping a robot understand the difference between "picking up a glass" and "crushing a glass" by focusing on the dynamics of the hand.
Search Engines: Allowing you to search for "a person jumping in slow motion" and actually getting the jump, not just a picture of a person in a gym.

In short: SemanticMoments teaches AI to stop looking at the actors and start watching the dance.

Technical Summary: SemanticMoments: Training-Free Motion Similarity via Third Moment Features

1. Problem Statement

The paper identifies a fundamental flaw in current video representation and retrieval models: appearance bias. Most existing methods (supervised, self-supervised, or multimodal) rely heavily on static visual cues—such as background, objects, and clothing—to identify actions. Consequently, they fail to capture true semantic motion (the structured temporal transformation of entities).

The authors observe two failure modes:

RGB/Multimodal Models: Over-rely on scene context (e.g., recognizing "playing cello" by seeing a cello, even if the motion is static).
Optical Flow Models: While robust to appearance, they lack semantic grounding, failing to understand high-level motion dynamics and struggling with changes in viewpoint or subject identity.

2. Methodology

The authors propose SemanticMoments, a training-free method that represents motion by computing higher-order temporal statistics over patch-level embeddings from pretrained semantic models (e.g., DINOv2, VideoMAE).

Core Technical Approach:

Feature Extraction: Instead of global frame pooling, the method extracts patch-level features ( $F_t \in \mathbb{R}^{P \times d}$ ) from each frame $t$ . This preserves spatial granularity.
Temporal Moments ( $M^+$ ): The method moves beyond the "first moment" (mean/average appearance) by calculating:
1. 1st Moment (Mean): Captures average appearance.
2. 2nd Moment (Variance): Captures the magnitude of temporal variation (motion energy).
3. 3rd Moment (Skewness): Captures the directional asymmetry or "polarity" of change (how motion unfolds).
Spatial Aggregation: Per-patch moments are aggregated spatially to create a global descriptor for each moment order.
Moment Embedding: The final video descriptor is a weighted concatenation of these moment vectors: $\phi_{video} = [\alpha_1 M^{(1)}; \alpha_2 M^{(2)}; \dots; \alpha_K M^{(K)}]$ .
Training-Free: The method requires no new training, no optical flow, and no labeled data, making it a plug-and-play module for any pretrained backbone.

3. Key Contributions

New Benchmarks (SimMotion):
- SimMotion-Synthetic: A controlled dataset using generative AI (WAN 2.2) to create video triplets where motion is identical but visual factors (viewpoint, style, object identity, etc.) vary. This isolates motion as the only variable.
- SimMotion-Real: A human-annotated dataset of real-world videos, focusing on perceptual motion similarity in unconstrained, "in-the-wild" scenarios.
SemanticMoments Framework: A mathematically grounded, efficient way to capture semantic dynamics using higher-order statistics.
Empirical Validation: Demonstrating that temporal statistics in a semantic feature space provide a scalable foundation for motion understanding.

4. Results

SimMotion-Synthetic: SemanticMoments consistently outperformed state-of-the-art baselines (CLIP4Clip, VideoMAE, VideoPrism, etc.). It showed particular strength in the "Dynamic Object" and "View" categories, where traditional models failed because they were sensitive to subject identity and camera angles.
SimMotion-Real: While real-world retrieval remains difficult for all models, SemanticMoments achieved significantly higher accuracy than existing RGB and multimodal methods, proving its robustness to natural variability.
Gesture Classification (Jester): Applying SemanticMoments to backbones like DINOv2 and VideoMAE improved kNN-based gesture recognition accuracy, confirming that the method enhances the separability of motion-based classes in the embedding space.
Ablation Studies: Confirmed that patch-level granularity and the inclusion of the 3rd moment (skewness) are critical for capturing fine-grained motion structure.

5. Significance

This work shifts the paradigm of video representation from "what is in the frame" to "how the semantic structure changes over time." By proving that higher-order temporal statistics can disentangle motion from appearance without expensive retraining, the paper provides a highly scalable path toward:

Motion-centric video retrieval (finding videos with similar actions regardless of subject).
Enhanced motion control for generative video models.
Improved dataset construction for training future motion-aware AI.