🤖 AI

Rebalancing Reference Frame Dominance to Improve Motion in Image-to-Video Models

This paper introduces DyMoS, a training-free and model-agnostic method that enhances motion dynamics in image-to-video generation by rebalancing reference-frame dominance in the attention mechanism, thereby overcoming motion suppression without compromising visual fidelity or requiring additional training.

Original authors: Wooseok Jeon, Seungho Park, Seunghyun Shin, Sangeyl Lee, Hyeonho Jeong, Hae-Gon Jeon

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wooseok Jeon, Seungho Park, Seunghyun Shin, Sangeyl Lee, Hyeonho Jeong, Hae-Gon Jeon

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Frozen" Video

Imagine you have a photo of a horse, and you ask a computer to turn it into a video of the horse running. You'd expect the horse to gallop. But often, current AI models are too "polite" to the original photo. They keep the horse frozen in place, maybe just wiggling its ears slightly, because they are so afraid of changing the picture you gave them.

The paper calls this the "Static Motion Bias." The AI is so focused on keeping the first frame (the reference image) perfect that it forgets to make the rest of the video move.

The Discovery: The "Over-Attentive" Student

The researchers looked under the hood of how these AI models work. They discovered a specific behavior they call "Reference-Frame Dominance."

Think of the AI model as a student taking a test.

The Text Prompt is the question: "Make the horse run."
The Reference Image is a picture of the horse on the desk.

In a normal Text-to-Video model, the student looks at the question and imagines the running horse.
In an Image-to-Video model, the student is staring too hard at the picture on the desk. Every time the student tries to draw the next frame of the video, they look at the picture and say, "Okay, I must make this next frame look exactly like the picture."

Because the student is so obsessed with copying the picture, they never actually draw the horse moving. The "picture" is dominating the "imagination."

The Solution: DyMoS (The "Motion Slider")

The researchers created a tool called DyMoS (Dynamic Motion Slider). It doesn't require retraining the AI or changing the original photo. Instead, it acts like a volume knob for the student's obsession.

How it works:

The Intervention: During the very first few moments of creating the video (the "denoising" steps), DyMoS gently taps the student on the shoulder. It says, "Hey, stop staring so hard at the reference picture. You need to imagine the motion now."
The Mechanism: Technically, it adjusts a specific number (a "bias") in the AI's attention system. It slightly lowers the score the AI gives to the reference image tokens when it's trying to decide what the next frame should look like.
The Result: The AI is still allowed to see the picture (so the horse still looks like the horse), but it's no longer forced to copy it perfectly. This frees up the AI to let the horse run, the car drive, or the water splash.

The "Slider" Feature

The coolest part of DyMoS is that it's a slider, not just an on/off switch. The user has one number they can turn:

Turn it down (Negative numbers): The AI becomes more obsessed with the picture. The video becomes even more static (useful if you want a still image that barely moves).
Turn it up (Positive numbers): The AI is told to ignore the picture's "frozen" nature more. The video becomes very dynamic and energetic.
Middle ground: You can find the perfect balance where the video moves naturally but the character still looks exactly like the photo you started with.

Why It's Better Than Previous Methods

Previous attempts to fix this "frozen video" problem were like trying to fix a car by smashing the engine or painting over the wheels.

Some methods required retraining the whole AI (expensive and slow).
Others tried to blur or mess up the input image to force motion, which often made the video look ugly or distorted.

DyMoS is different because:

It's Training-Free: You don't need to teach the AI anything new. You just use the tool while the AI is working.
It's Model-Agnostic: It works on almost any modern video AI (like Wan 2.2, HunyuanVideo, CogVideoX).
It Keeps Quality: It makes the video move without making the picture look blurry or weird.

Summary

The paper identifies that Image-to-Video models are "too polite" to their input images, causing videos to be boring and still. They fixed this with DyMoS, a simple tool that acts like a volume knob, dialing down the AI's obsession with the starting photo just enough to let the motion happen, all without changing the model or the original image.

Technical Summary: Rebalancing Reference Frame Dominance to Improve Motion in Image-to-Video Models

Problem Statement

Image-to-video (I2V) models, while capable of preserving the appearance of a reference image, frequently suffer from a "static motion bias." Compared to text-to-video (T2V) counterparts, I2V models often generate videos with overly limited or weak motion, even when the text prompt explicitly describes dynamic action. Existing approaches to mitigate this issue typically involve modifying the image-conditioning pathway (e.g., adding noise or redesigning flow matching) or weakening the conditioning signal. However, these methods often require additional training or introduce a trade-off where improved motion comes at the cost of fidelity to the reference image.

Methodology: DyMoS (Dynamic Motion Slider)

The authors propose DyMoS, a training-free and model-agnostic method that addresses static motion bias by intervening directly in the self-attention mechanism during inference, without altering model weights or the input image.

Core Insight: Reference-Frame Dominance

Through an analysis of self-attention maps in paired T2V and I2V generations, the authors identify a phenomenon they term reference-frame dominance. In I2V models, query tokens in non-reference frames allocate significantly more attention to the key tokens of the reference frame (the first frame) compared to their T2V counterparts. This excessive attention propagates reference information across the temporal axis, suppressing inter-frame dynamics and resulting in static outputs.

Technical Implementation

DyMoS mitigates this dominance by rebalancing the attention logits during the initial denoising steps. The method introduces a scalar bias shift to the attention logits associated with reference-frame key tokens.

The modulation is applied as follows:

Logit Adjustment: For non-reference query tokens ( $i \notin I_{f0}$ ) attending to reference key tokens ( $j \in I_{f0}$ ), a positive scalar $\gamma$ is subtracted from the attention logits $L[i, j]$ before the softmax operation:
$\tilde{L}[i, j] = L[i, j] - \gamma \cdot \phi(f(i)) \cdot \mathbb{1}[j \in I_{f0}] \cdot \mathbb{1}[i \notin I_{f0}]$
Here, $\gamma$ controls the strength of the intervention, and $\phi(\cdot)$ is a frame-wise schedule (defaulting to uniform).
Temporal Scope: The modulation is applied only during the first $\lambda$ fraction of the denoising steps. This targets the early stage where global motion is determined, while allowing later steps to refine appearance and temporal coherence using standard attention.
Classifier-Free Guidance (CFG): The modulation is applied exclusively to the text-conditional branch of the CFG equation, leaving the null-text branch unchanged. This steers the guided update away from reference-frame dominance while maintaining the structural integrity of the generation process.

Key Contributions

Identification of Mechanism: The paper identifies "reference-frame dominance" in self-attention as a primary driver of static motion in I2V models, distinguishing it from simple input conditioning issues.
Training-Free Intervention: DyMoS offers a simple, model-agnostic solution that requires no fine-tuning, retraining, or modification of the input image. It operates solely by adjusting attention logits during inference.
Continuous Control: The method introduces a single scalar parameter ( $\gamma$ ) that allows users to continuously control the spectrum from strict reference-image preservation (static) to highly dynamic motion, effectively bridging the gap between I2V and T2V motion dynamics.
Preservation of Fidelity: Unlike prior methods that sacrifice visual quality or reference consistency to enhance motion, DyMoS improves motion dynamics while maintaining high fidelity to the reference image and visual quality.

Experimental Results

The authors validated DyMoS across multiple state-of-the-art I2V backbones, including Wan 2.2, Wan 2.1, HunyuanVideo-1.5, and CogVideoX-5B.

Motion Dynamics: Across all backbones, DyMoS consistently achieved higher Dynamic Degree scores (measured by VBench and VideoScore) compared to both the vanilla baseline and the state-of-the-art method ALG. For instance, on Wan 2.2, the Dynamic Degree increased from 51.7 (Baseline) to 64.8 (DyMoS), a significant improvement over ALG's 55.3.
Visual Quality: Crucially, DyMoS maintained Video Quality scores comparable to the baseline and ALG, demonstrating that enhanced motion does not degrade visual fidelity.
Text Alignment: The method achieved the highest ViCLIP scores among compared methods, suggesting that reducing reference-frame dominance allows the model to better follow motion descriptions in text prompts.
Human Preference: In a user study, DyMoS was selected as the best video across all criteria (Motion Dynamicness, Reference-Frame Fidelity, Text Alignment, and Overall Preference) more frequently than the baseline or ALG.
Efficiency: The method incurs negligible inference-time overhead (e.g., 1.2% on Wan 2.2), as it involves only lightweight element-wise operations on attention logits rather than additional forward passes.

Significance and Claims

The paper claims that the static motion bias in I2V generation is not an inherent limitation of the conditioning mechanism itself, but rather a consequence of how reference information is routed through self-attention. By intervening at the attention-logit level, DyMoS effectively rebalances the model's focus, enabling dynamic motion without compromising the core strengths of I2V models (reference fidelity).

The authors position DyMoS as a practical tool for users and developers, offering a "sliding scale" of motion control via a single parameter. They note that while the method is effective, overly large modulation strengths can reduce appearance consistency, and future work could explore automatic adaptation of modulation strength based on specific prompts or motion types. The work demonstrates that simple, targeted interventions in the attention mechanism can yield substantial improvements in generative video quality without the computational cost of retraining.