💻 computer science

Tempered Self-Similarity Alignment for Physically Plausible Video Generation

This paper introduces Tempered Self-similarity Alignment (TSA), a novel training method that enhances the physical plausibility of generated videos by aligning their spatio-temporal self-similarity distributions with those of visual foundation models to better capture real-world object interactions and dynamics.

Original authors: Manjin Kim, Suha Kwak, Minsu Cho

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Manjin Kim, Suha Kwak, Minsu Cho

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a talented but inexperienced artist how to paint a moving scene, like a hand pushing a toy truck. The artist (the video generation model) is great at making things look pretty, but they often mess up the physics. They might make the truck slide like it's on ice when it should be rolling, or make the hand disappear and reappear in weird spots. This is called "appearance drift" and "unrealistic motion."

The researchers behind this paper, Tempered Self-Similarity Alignment (TSA), came up with a clever way to fix this by hiring a strict, expert art critic (a "visual foundation model") to guide the artist.

Here is how their method works, broken down into simple concepts:

1. The Problem: The Artist vs. The Expert

The Artist (Video Model): When the artist tries to draw a sequence of frames, they get the "vibe" right but miss the details. If a ball bounces, the artist might make it look like it's floating or changing shape strangely.
The Expert (Foundation Model): This is a super-smart AI that has seen millions of videos. It doesn't just see colors; it understands the relationships between objects. It knows that if a hand moves, the toy truck must move with it in a specific way. It sees the "story" of how things connect across time.

2. The Solution: The "Self-Similarity" Map

The researchers realized that the best way to teach the artist isn't to show them the final picture, but to show them a map of connections.

What is Self-Similarity? Imagine you take a snapshot of a video and ask, "If I look at this specific pixel on the toy truck in frame 1, where does it go in frame 2? Frame 3?"
The Expert AI creates a map showing these connections. It says, "This pixel on the truck in the first frame is strongly connected to that pixel in the second frame."
The Artist's map is usually messy and blurry. The goal is to make the Artist's map look exactly like the Expert's map.

3. The Secret Sauce: "Tempering" (Sharpening the Focus)

The researchers found that if they just told the Artist to copy the Expert's map, the Artist would get confused because the map was too "smooth" and vague. It was like giving directions that said, "Go somewhere near the park."

The Fix (Tempering): They applied a mathematical trick called "tempering." Think of this like turning up the contrast on a photo or sharpening a blurry image.
The Result: Instead of a vague "maybe here," the map becomes a sharp, clear arrow pointing exactly where the object should go. It turns a fuzzy guess into a precise instruction: "This specific part of the truck must move to this specific spot." This helps the Artist learn the fine details of motion.

4. The Efficiency Trick: Ignoring the Background

Imagine you are trying to learn how to juggle. If your teacher keeps pointing at the wall, the floor, and the ceiling, you get distracted. You only care about the balls and your hands.

The Problem: Most of a video is actually boring static stuff (a wall, a sky, a table). The Expert AI spends energy calculating connections for these static parts, which confuses the Artist.
The Fix (Masking): The researchers told the system to ignore the static background. They put a "mask" over the boring parts and only let the Artist focus on the moving parts (the dynamic regions).
The Benefit: This saves energy and forces the Artist to pay attention only to where the physics actually matter: the moving objects.

5. The Results: A More Realistic World

When they tested this new method on two major tests (VideoPhy and VideoPhy2), the results were impressive:

Before: Videos often looked like magic tricks where objects floated, melted, or teleported.
After: The videos looked like real life. When a hand pushed a toy, the toy rolled. When a boat hit water, the splash happened after the boat passed, not before.

In Summary

The paper introduces a system that takes the "intuition" of a super-smart AI (which understands how objects relate to each other over time) and forces a video generator to copy that intuition. By sharpening the instructions (tempering) and ignoring the boring background (masking), they taught the video generator to create movies that obey the laws of physics, making the motion look natural and believable.

Technical Summary: Tempered Self-Similarity Alignment for Physically Plausible Video Generation

Problem Statement
Despite significant advancements in video generative models, particularly text-to-video diffusion models, they frequently fail to produce physically realistic videos. Common failure modes include appearance drift, unrealistic deformations, and inconsistent motion dynamics. These limitations hinder the application of video generation in domains requiring accurate motion simulation, such as robotics training and virtual environment construction. Existing approaches to improve physical realism face distinct challenges: physics simulators are computationally heavy and domain-specific; multimodal large language models (MLLMs) rely on iterative refinement that increases latency and is bounded by their reasoning capabilities; and optical flow signals primarily capture short-term displacements, struggling to supervise long-range dynamics or structural changes.

Methodology
The authors propose Tempered Self-Similarity Alignment (TSA), a framework designed to transfer relational knowledge encoded in Spatio-Temporal Self-Similarity (STSS) from pre-trained visual foundation models (e.g., DINOv3) into video generative models (e.g., CogVideoX).

Spatio-Temporal Self-Similarity (STSS): The method leverages STSS, which represents pairwise similarities among visual features across space and time. Unlike raw appearance features, STSS suppresses photometric variations (color, texture) to emphasize relational structures, effectively capturing how objects interact, move, and deform throughout a video.
Tempered Alignment: Instead of directly aligning raw STSS tensors (which often yields overly smooth, ambiguous correspondence distributions), the authors introduce a tempering mechanism. They apply temperature-scaled normalization to transform STSS into sharpened probabilistic correspondence distributions.
- The visual foundation model and the video diffusion model both generate feature maps.
- Pairwise cosine similarity matrices are computed for both.
- A temperature parameter ( $\tau$ ) is applied before a softmax operation to sharpen the correspondence probability distributions. Lowering $\tau$ shifts the granularity from object-level to part-level or point-level correspondences.
- The TSA loss minimizes the Kullback-Leibler (KL) divergence between the correspondence distributions of the foundation model (teacher) and the diffusion model (student). This provides stronger gradients for misaligned regions compared to uniform gradient approaches.
Motion-Focused Masking (M-TSA): Recognizing that most video regions are static, the authors introduce a Motion-Saliency Mask. This mask identifies dynamic regions by calculating temporal differences across frames and selecting the top- $k$ tokens with the largest variations. The alignment loss is restricted exclusively to these dynamic regions, reducing computational overhead and preventing the model from being biased toward static backgrounds.

Key Contributions

TSA Loss: A novel loss function that transfers relational knowledge from visual foundation models to video diffusion models by aligning probabilistic spatio-temporal correspondence distributions.
Tempering Mechanism: A technique that transforms STSS into sharpened distributions via temperature scaling, enabling more precise, fine-grained motion supervision than direct tensor alignment.
Dynamic Region Restriction: The introduction of background masking (M-TSA) that focuses supervision on motion-salient areas, improving learning efficiency and physical grounding.

Experimental Results
The method was evaluated on VideoPhy and VideoPhy2 benchmarks, which assess physical plausibility and semantic adherence in diverse interaction scenarios (solid-solid, solid-fluid, fluid-fluid, and human-object interactions).

VideoPhy: The proposed method (LM-TSA) achieved a Physical Commonsense (PC) score of 30.8%, outperforming the baseline CogVideoX-2B (25.0%) and the state-of-the-art VideoREPA (27.6%). The most significant improvement was observed in solid-solid interactions, where the PC score rose from 15.4% to 21.0%.
VideoPhy2: On the human-object interaction benchmark, the method achieved a Joint score (SA $\ge$ 4 and PC $\ge$ 4) of 24.4%, surpassing previous methods.
VBench: Evaluation on VBench confirmed that the method enhances physical plausibility without compromising overall video quality or text-condition consistency.
Ablation Studies: Experiments demonstrated that an optimal temperature ( $\tau = 0.1$ ) balances fine-grained motion cues with structural context, and a masking ratio of $k=20\%$ yields the best performance by filtering out redundant static regions. The approach was also validated on the NOVA auto-regressive model, demonstrating architectural generality.

Significance and Claims
The paper claims that TSA effectively addresses the limitation of current video generative models in producing physically plausible content by transferring relational knowledge regarding object dynamics. By converting STSS into sharpened probabilistic distributions and restricting alignment to dynamic regions, the method provides precise motion supervision that captures long-range dynamics and structural transformations better than optical flow or raw feature alignment. The authors assert that this approach validates the effectiveness of transferring relational knowledge from foundation models to generative models, achieving state-of-the-art performance in generating physically realistic videos across diverse scenarios.