💻 computer science

Durian: Dual Reference Image-Guided Portrait Animation with Attribute Transfer

Durian is a novel portrait animation method that enables cross-identity attribute transfer from one or more reference images by employing a self-reconstruction training strategy with a Dual ReferenceNet and complementary masking to learn from unpaired video data, achieving state-of-the-art performance with support for multi-attribute composition and smooth interpolation.

Original authors: Hyunsoo Cha, Byungjun Kim, Hanbyul Joo

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hyunsoo Cha, Byungjun Kim, Hanbyul Joo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a photo of yourself, but you're bored with your current look. You want to try on a cool new hat, swap your hair for a wild afro, or add a pair of stylish sunglasses—all while you are talking, smiling, and moving your head naturally.

Usually, doing this is a nightmare for computers. To teach a computer how to do this, you'd typically need thousands of photos of the same person wearing every possible combination of hats, glasses, and hairstyles. That's impossible to collect.

Enter Durian (the new AI method from Seoul National University). Think of Durian as a magical, super-smart video editor that can learn to swap your look without ever needing a "before and after" photo of the same person.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Missing Puzzle Piece"

Normally, AI learns by looking at pairs: "Here is Person A with short hair, and here is Person A with long hair." It learns to swap the hair. But we don't have millions of videos of the same person changing their hair every day.

The Durian Solution: Instead of waiting for perfect pairs, Durian learns by playing a game of "self-reconstruction" using random videos of people talking.

It grabs two random frames from the same video of a person.
It treats one frame as the "Target" (the person's face).
It treats the other frame as the "Reference" (the new hair or glasses).
It tries to rebuild the video, forcing itself to figure out: "How do I take the hair from Frame B and put it onto the face in Frame A, while keeping the face looking like Frame A?"

It's like learning to cook a new dish by tasting two different soups and guessing the recipe, rather than having the recipe book in front of you.

2. The Secret Sauce: The "Dual Reference" Kitchen

Most AI models have one brain to look at the face and one to look at the new hair. Durian has a Dual ReferenceNet, which is like a kitchen with two specialized chefs:

Chef Identity (PRNet): This chef only looks at the face and says, "I must keep the eyes, nose, and smile exactly the same."
Chef Attribute (ARNet): This chef only looks at the new hair or glasses and says, "I must copy this style perfectly."

These two chefs work separately but talk to each other constantly through a Spatial Attention mechanism. Think of this as a high-tech laser pointer. The laser tells the AI exactly where on the face the new hair should go and where the glasses should sit, ensuring they don't accidentally paste the hair over the eyes!

3. The Training Trick: "Masking" and "Stretching"

To make sure the AI doesn't get lazy (like just copying the whole face from the reference), Durian uses masks.

It covers up the hair on the reference photo so the AI only sees the hair.
It covers up the hair on the target photo so the AI must generate new hair there.

But here's the clever part: Mask Expansion.
Imagine you are teaching a child to draw a hat. If you only show them a small, tight hat, they might struggle if you ask them to draw a big floppy one later. Durian artificially "stretches" the training masks. It teaches the AI that a "hat" isn't just one specific shape; it can be big, small, tilted, or shifted. This makes the AI robust, meaning it won't break if the new glasses are slightly crooked or the new hair is wilder than the reference.

4. The Magic at the End: Mixing and Matching

Because Durian is so smart, it doesn't just do one thing at a time.

The "Smoothie" Effect (Interpolation): You can ask Durian to blend two hairstyles. It can create a video where your hair slowly morphs from a short bob to a long ponytail, frame by frame.
The "Combo Meal" (Multi-Attribute): You can give it a photo of a hat, a photo of glasses, and a photo of a beard. Durian can combine all three into a single video instantly. It knows that the hat goes over the hair and the glasses go on the nose, handling the overlaps perfectly.

Why is this a big deal?

Before Durian, if you wanted to try on a new hairstyle in a video, you needed a 3D scanner, a specific app, or a pre-made template that looked stiff and fake.

Durian is the "Zero-Shot" Wizard.

Zero-Shot: It doesn't need to be retrained for every new person or new object.
In-the-Wild: It works with random photos and videos you find on the internet.
One-Pass: It does the swapping and the animation in one go, like a single magic spell, rather than a long, complicated process.

In short: Durian is the first AI that can look at a photo of a stranger's cool haircut and a video of you, and seamlessly edit your video to have that haircut, all while you talk and move, without ever needing a dataset of you changing your hair. It's like having a personal stylist who can instantly try on any look from any photo, right on your face.

1. Problem Statement

The paper addresses the challenge of generating portrait animation videos with cross-identity attribute transfer. Specifically, the goal is to take a target portrait image and one or more reference images (specifying attributes like hairstyle, glasses, hats, or beards) and generate a video where the target person's identity is preserved while their facial attributes are replaced or modified, all driven by a sequence of facial keypoints.

Key Challenges:

Data Scarcity: Training such models typically requires paired data (the same person with different attributes), which is rare and expensive to collect at scale, especially for multiple attribute combinations.
Identity vs. Attribute Disentanglement: Existing methods often struggle to separate the target's identity from the source attributes, leading to "identity leakage" (where the target looks like the reference person) or poor attribute fidelity.
Limitations of Current Solutions: Most existing tools are specialized (e.g., only for hair or only for glasses), rely on fixed 3D templates, or require multi-stage pipelines (image editing followed by animation) that lack temporal consistency.

2. Methodology

The authors propose Durian, a diffusion-based generative framework that learns attribute transfer without explicit paired data using a self-reconstruction strategy.

A. Self-Reconstruction Training Framework

Instead of requiring paired data (Person A with Attribute X, Person A with Attribute Y), Durian learns from standard "in-the-wild" portrait videos:

Pseudo-Pair Generation: Two frames are randomly sampled from a single video of one person. One frame serves as the Attribute Reference ( $I_{attr}$ ), and the other as the Identity Reference ( $I_{port}$ ).
Complementary Masking: To prevent the model from simply copying the whole image (identity leakage), the inputs are masked:
- Attribute-only input: $I_{attr} \odot M_{attr}$ (keeps only the attribute region, e.g., hair).
- Attribute-masked portrait: $I_{port} \odot (1 - M_{port})$ (removes the corresponding region from the target).
Objective: The model is trained to reconstruct the original video frames conditioned on these masked inputs and a facial keypoint sequence. This forces the network to learn how to inject the attribute from the first frame into the identity of the second frame.

B. Dual ReferenceNet Architecture

The core innovation is the Dual ReferenceNet, which consists of two separate encoder branches:

Attribute ReferenceNet (ARNet): Encodes the attribute-only image.
Portrait ReferenceNet (PRNet): Encodes the attribute-masked portrait.
Spatial Attention Fusion: Instead of concatenating features, the model uses a Spatial Attention mechanism to fuse features from ARNet and PRNet into the main Denoising U-Net (DNet). This allows the model to attend to specific spatial regions of the attribute reference while preserving the global structure of the identity reference.
Semantic Embeddings: CLIP embeddings are used for the attribute, and a combination of ArcFace and CLIP embeddings are used for the portrait to enhance identity preservation.

C. Training Strategies for Robustness

To bridge the gap between self-reconstruction (same identity) and cross-identity inference (different identities), the authors introduce:

Attribute-Aware Mask Expansion: During training, the attribute mask is expanded using generative augmentation (SDXL + ControlNet) to simulate different shapes and spatial extents of the attribute. This prevents the model from overfitting to the specific shape of the attribute in the training video.
Reference Image Augmentation: Random affine transformations (rotation, scaling) and color jittering are applied to the reference images to simulate misalignment and lighting variations.
Inference Alignment: At inference, a lightweight Face Aligner (using 3D avatar reconstruction) aligns the attribute reference to the target portrait before generating the mask, ensuring spatial consistency.

D. Inference Capabilities

Multi-Attribute Composition: By concatenating features from multiple ARNets, the model can transfer multiple attributes (e.g., hair + glasses + hat) in a single pass without retraining.
Attribute Interpolation: By linearly blending the spatial features of two attribute references, the model can generate smooth transitions between attributes (e.g., short hair to long hair).

3. Key Contributions

First Zero-Shot Cross-Identity Animation: The first method to generate keypoint-driven portrait animations with attribute transfer from uncurated videos without explicit attribute-paired data.
Dual ReferenceNet: A novel architecture that disentangles identity and attribute streams via separate encoders and fuses them via spatial attention, enabling effective self-reconstruction training.
Robustness Mechanisms: The introduction of mask expansion and augmentation strategies effectively bridges the domain gap between self-reconstruction training and cross-identity inference.
Emergent Capabilities: The framework naturally supports multi-attribute composition and smooth attribute interpolation in a single generation pass without additional training.

4. Experimental Results

The authors evaluated Durian on CelebV-Text, VFHQ, and Nersemble datasets.

Baselines: Compared against 12 combinations of state-of-the-art image editing (e.g., Paint-by-Example, StableHair) and video animation models (e.g., LivePortrait, X-Portrait).
Quantitative Performance: Durian achieved State-of-the-Art (SOTA) results in both self-attribute and cross-attribute transfer settings.
- Self-Attribute: Outperformed baselines in L1, PSNR, SSIM, and LPIPS.
- Cross-Attribute: Showed superior identity preservation (ArcFace similarity) and attribute transfer quality (CLIP-I, DINO) while maintaining temporal realism (VFID).
Qualitative Results: Visual comparisons showed Durian generates more coherent animations with better preservation of fine details and identity compared to baselines, which often suffered from identity leakage or inconsistent attribute shapes.
Ablation Studies: Confirmed that the Dual ReferenceNet, mask expansion, and reference image augmentation are critical components; removing them led to identity leakage or poor handling of diverse attribute shapes.

5. Significance

Durian represents a significant leap in personalized digital styling and virtual avatars.

Scalability: By eliminating the need for expensive, manually curated paired datasets, it opens the door to training on vast amounts of existing video data.
Flexibility: Its ability to handle multiple attributes simultaneously and interpolate between them offers a highly controllable tool for creative applications.
Practicality: The zero-shot nature allows users to apply any attribute (hair, glasses, beard) from any reference image to any target portrait instantly, making it highly suitable for real-world virtual try-on and content creation applications.