SAGE: Structure-Aware Generative Video Transitions between Diverse Clips

SAGE is a zero-shot, structure-aware generative framework that synthesizes visually coherent and motion-consistent video transitions between diverse clips by combining line maps and motion flow guidance with generative synthesis, effectively outperforming existing classical and generative methods without requiring fine-tuning or specific training data.

The Gist

Imagine you are a film editor trying to stitch together two completely different movie scenes. Maybe you want to transition from a sunny beach to a bustling city street, or from a galloping horse to a racing car.

If you just use a standard "fade" (like slowly turning down the volume on one scene while turning it up on the other), the result looks messy. It's like trying to blend a cup of coffee with a bowl of soup; you get a muddy, confusing mess with "ghosts" of both scenes floating around.

This is the problem SAGE (Structure-Aware Generative vidEo transitions) solves. It's a new tool that acts like a magical, invisible bridge builder between two very different video clips.

Here is how it works, explained through simple analogies:

1. The Problem: The "Ghostly" Fade

Current methods try to guess what happens in between two clips.

• The Old Way: It's like taking a photo of a horse and a photo of a car, then slowly fading one into the other. The result is a blurry, confusing mess where the horse's legs might turn into car wheels, or the background just dissolves into static.
• The Result: The video "breaks." The motion feels jerky, objects disappear and reappear randomly, and the transition feels fake.

2. The SAGE Solution: The "Architect's Blueprint"

Instead of just guessing, SAGE acts like a smart architect who draws a blueprint before building a bridge. It looks at the two clips and asks: "How can we walk from here to there without falling off a cliff?"

It does this in three clever steps:

Step A: Finding the "Skeleton" (Structural Anchoring)

Imagine you are trying to morph a picture of a cat into a dog. If you just blend the pixels, the cat's ears might turn into a dog's nose.

• What SAGE does: It ignores the fur and colors for a moment. Instead, it looks for the outline or the "skeleton" of the shapes. It finds the lines that define the cat's back and the dog's back.
• The Analogy: Think of it like tracing the outline of a shadow. SAGE traces the "shadow" of the important objects in both clips so it knows exactly where the main shapes are.

Step B: Drawing the "Road" (Motion Continuity)

Just knowing where the objects are isn't enough; you need to know how they move.

• The Problem: If the camera in the first clip is moving left, and the second clip is moving right, a simple blend creates a chaotic spin.
• What SAGE does: It draws a smooth, curved road (called a B-spline) that connects the start point to the end point. It ensures the "road" follows the natural flow of the video.
• The Analogy: Imagine driving a car. If you just turn the steering wheel randomly, you crash. SAGE calculates the smoothest, most logical curve to drive from the beach to the city, ensuring the car (the video) never swerves wildly.

Step C: The "Layered Cake" (Layered Blending)

Artists know that you don't change everything at once.

• What SAGE does: It separates the foreground (the main actors, like a surfer) from the background (the ocean).
• The Analogy: Think of a cake. SAGE changes the frosting (background) slowly and gently, while carefully reshaping the fruit on top (the foreground) to match the new scene. This prevents the "ghosting" effect where objects seem to double up or vanish.

3. The Magic Trick: Zero-Shot Learning

Usually, to teach a computer to do something new, you need to show it thousands of examples (like showing a student 1,000 pictures of bridges so they learn to build one).

• The Challenge: There are no databases full of "perfect transitions" between a horse and a car.
• The SAGE Trick: SAGE doesn't need to be retrained. It takes a pre-trained AI (which is already good at making videos) and gives it the blueprint (the lines and motion paths) we drew in Step A and B.
• The Result: The AI says, "Oh, you want me to build a bridge from A to B? Here is your blueprint. I will fill in the details!" It creates a smooth, realistic transition instantly, without needing to learn from scratch.

Why is this a big deal?

• For Filmmakers: It allows them to create "match cuts" (where one scene morphs perfectly into another) that were previously impossible without hours of manual editing.
• For Creators: It makes video editing feel like magic. You can take a clip of a dancer and a clip of a rocket, and SAGE will invent the frames in between where the dancer becomes the rocket, moving smoothly and logically.

In short: SAGE is like a smart translator for video. It doesn't just translate words (pixels); it understands the grammar (structure) and the story (motion) to create a seamless sentence between two completely different ideas.

Technical Summary

1. Problem Statement

The paper addresses the challenge of generating video transitions between two diverse clips (clips that differ significantly in content, style, motion, or semantics).

• Limitations of Existing Methods:
  • Traditional Techniques: Linear blending (cross-fades) and standard morphing often result in ghosting, flickering, or structural collapse because they ignore scene content.
  • Generative Inbetweening: Recent diffusion-based methods (e.g., DiffMorpher, Generative Inbetweening) can hallucinate plausible intermediates but typically assume small temporal gaps and semantic consistency. They struggle when bridging clips with large visual or motion differences, often failing to maintain temporal coherence or producing spurious objects.
• The Gap: There is a lack of zero-shot methods that can generate content-aware, visually coherent, and motion-consistent transitions for diverse clips without requiring massive, curated training datasets (which are scarce for artistic transitions).

2. Methodology: SAGE

SAGE (Structure-Aware Generative vidEo transitions) is a zero-shot approach that fuses structural guidance with generative synthesis. It is inspired by artist workflows, distilling three key heuristics: Structural Anchoring, Motion Continuity, and Layered Blending.

The algorithm operates in three stages (see Figure 3 in the paper):

Stage I: Feature Extraction

Given the boundary frames of two clips ($C_A$ and $C_B$), SAGE extracts:

1. Structural Features: Line segments representing silhouettes and dominant contours using a pretrained line detector (GlueStick).
2. Motion Features: Optical flow fields using SEA-RAFT to capture local motion cues.
3. Layer Features: Foreground masks (using SAM) to isolate salient objects from the background.

Stage II: Motion-Aware Structural Interpolation

Instead of naive linear blending, SAGE interpolates structural primitives using a hierarchical scheme:

1. Layer-Aware Line Matching:
   • Restricts matching to foreground lines to avoid background clutter dominating the transition.
   • Applies canonical normalization (bounding box alignment) to handle scale and position differences.
   • Uses Hungarian matching to establish one-to-one correspondences between foreground lines in $C_A$ and $C_B$.
2. Motion-Aware B-Spline Propagation:
   • Global Trajectory: Computes a global foreground trajectory using cubic B-splines based on average flow vectors and bounding box displacements. This ensures the transition follows a smooth, motion-aware path, preventing unnatural jumps.
   • Local Interpolation: Matches lines are blended linearly within the local canonical space defined by the global B-spline trajectory. This couples local geometric evolution with global scene dynamics, avoiding trajectory crossings and structural collapse.

Stage III: Conditional Generative Synthesis

The interpolated line sets are rasterized into edge maps ($E_t$) and used as conditions for a pretrained diffusion-based inbetweening model (specifically, the model from [ZRW∗25]).

• The model takes the start frame, end frame, and the intermediate edge maps as input.
• This allows the model to synthesize temporally smooth frames that adhere to the structural and motion priors without fine-tuning.

3. Key Contributions

1. Zero-Shot Framework: SAGE is the first method to enable structure-aware generative transitions between diverse clips without task-specific fine-tuning or curated training data.
2. Hierarchical Structural Anchoring: Introduces a novel pipeline that extracts, normalizes, and matches salient line structures in a layer-wise manner to prevent background dominance and ensure semantic alignment.
3. Motion-Aware B-Spline Propagation: Proposes a two-scale interpolation scheme (global B-spline trajectory + local line blending) that mitigates trajectory crossings and incoherent motion common in naive interpolation.
4. Integration of Guidance: Successfully unifies geometric/motion guidance with generative synthesis, leveraging pretrained models to achieve high-quality results where they previously failed.

4. Results and Evaluation

The authors evaluated SAGE against classical baselines (cross-fade) and state-of-the-art generative methods (FILM, SEINE, DiffMorpher, TVG, VACE, Generative Inbetweening).

• Quantitative Metrics:
  • Flow Similarity: SAGE achieved the highest flow similarity (0.69) compared to the ground truth, significantly outperforming baselines (e.g., DiffMorpher at 0.61, GI at 0.55). This indicates superior motion consistency.
  • Video Quality (FVD): SAGE achieved competitive results (2185), second only to TVG (2093), while maintaining better motion coherence.
  • Image Quality (FID): SAGE scored 153, comparable to other strong baselines.
• User Study:
  • In a study with 26 participants, SAGE was strongly preferred over all baselines across all criteria: Temporal Consistency (81.57% avg preference), Plausibility, Motion Complexity, and Overall Preference.
  • Users noted that SAGE produced smoother, more natural transitions, whereas baselines often suffered from ghosting, structural collapse, or abrupt mid-transition jumps.
• Qualitative Analysis:
  • SAGE successfully handled diverse scenarios including large scale changes, different object categories (e.g., Horse $\to$ Dog), and distinct motion directions (e.g., Skatepark Biker).
  • Baselines often failed by introducing unrelated objects (hallucinations) or creating "wiping" artifacts that ignored actual motion direction.

5. Significance and Future Work

• Significance: SAGE bridges the gap between artistic manual transition design and automated generative AI. It provides a practical tool for video editors to create professional-grade transitions between disparate clips without the need for expensive data collection or model retraining.
• Limitations: The method relies on line detection and optical flow, which can fail in texture-less regions or with occlusions. It also relies on a backbone pretrained on human poses, occasionally causing "human-like" hallucinations in non-human contexts.
• Future Directions: The authors suggest incorporating semantic cues (e.g., Dino features) for better matching, using higher-order correspondence costs, and integrating appearance-aware generation to handle texture-rich regions.

In conclusion, SAGE demonstrates that explicit structural guidance is crucial for controlling generative video models in zero-shot settings, enabling the synthesis of complex, coherent transitions that purely data-driven approaches cannot currently achieve.