💻 computer science

Loki: Representation over Architecture for Diffusion-Based Portrait Animation

Loki introduces a novel diffusion-based portrait animation framework that replaces RGB-based conditioning with an identity-orthogonal parametric face model and lightweight key-value injection, achieving superior pose and expression control with significantly fewer parameters and training data while enabling zero-shot cross-ID reenactment.

Original authors: Pouyan Navard, Sernam Lim

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pouyan Navard, Sernam Lim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Changing the "Language" of Animation

Imagine you want to take a photo of a friend (the Reference) and make them speak and move like a different person in a video (the Driver). This is called "Portrait Animation."

Most current AI systems try to do this by looking at the video and the photo as pixels (tiny colored dots). The problem is that pixels are messy. In a pixelated image, the person's face shape, their smile, and the way they turn their head are all mixed together in the same pile of dots. To separate them, the AI has to learn complex, expensive tricks to "unmix" the identity from the movement. It's like trying to separate a blended smoothie back into whole fruit, milk, and ice just by looking at the liquid.

Loki takes a completely different approach. Instead of looking at pixels, it looks at the blueprint of the face.

The Core Innovation: The "Face Blueprint" (FLAME)

The authors use a tool called FLAME, which is a 3D mathematical model of a human face. Think of FLAME not as a picture, but as a digital clay sculpture with specific knobs and dials:

Knob A: Controls the shape of the face (Identity).
Knob B: Controls the smile, frown, or jaw drop (Expression).
Knob C: Controls turning the head left or right (Pose).

In most other systems, these knobs are tangled. In Loki, they are perfectly separate. You can turn the "Smile" knob without accidentally changing the "Face Shape" knob.

How Loki Works: The Two-Track System

Loki uses a "diffusion" model (a type of AI that generates images from noise). Usually, these models need a lot of heavy machinery to understand what to draw. Loki simplifies this by splitting the instructions into two distinct tracks:

1. The Driver Track (The "Motion Map")
Instead of feeding the AI the video of the driver, Loki takes the driver's video, extracts the FLAME knobs (how much they smiled, how much they turned), and turns those numbers into a special map.

The Analogy: Imagine a topographic map of a mountain. The lines on the map don't tell you who is standing on the mountain; they just tell you where the peaks and valleys are and how steep they are.
Loki creates a map that says, "Here is a smile," and "Here is a head turn," but it contains zero information about who the person is. It's purely about the movement.

2. The Identity Track (The "Face Mask")
The AI then takes the photo of the person you want to animate (the Reference). It feeds this photo into the system to learn what the face looks like.

The Analogy: This is like taking a blank clay mold and pressing the Reference's face into it to get the shape.

The Magic Trick:
Because the "Motion Map" (Driver) has no identity in it, and the "Face Mold" (Reference) has no movement in it, Loki can simply swap the maps.

It takes the movement from the driver's map.
It applies it to the shape of the reference's face.
Result: The reference person moves exactly like the driver, but keeps their own face.

Why This is a Big Deal

1. It's Cheaper and Faster
Because the AI doesn't have to learn how to untangle identity from movement (since the blueprint already separates them for it), it needs 43% fewer parameters (less brain power) and 1,496 times less video data to train.

Analogy: Other systems are like a student trying to learn a language by memorizing every sentence in a dictionary. Loki is like giving the student a grammar book that already explains the rules perfectly. They learn much faster.

2. It Works on Strangers (Cross-ID)
Usually, to make a system work well when the driver and the reference are different people, you need to train it on thousands of examples of different people acting together.

Loki's Secret: Because the math of the blueprint is so clean, Loki can learn on just one person acting, and then instantly work on any other person. It's like learning to ride a bike on a training wheel, and then immediately being able to ride a mountain bike without ever having practiced on one. No extra training needed.

3. Better Accuracy
The paper introduces two new ways to measure success. Instead of just asking, "Does the picture look sharp?" (which is what old methods check), they ask: "Did the head turn the exact same amount?" and "Did the mouth open the exact same width?"

Loki wins at these specific tasks. It captures big, dramatic movements (like a wide-open mouth or a sharp head turn) much better than previous systems, which often make the movements look "meh" or generic.

Summary

Loki is a new way to animate faces that stops trying to "guess" the movement from a blurry video. Instead, it uses a mathematical blueprint that naturally separates "who the person is" from "what they are doing."

Old Way: Try to unmix a smoothie to find the fruit. (Hard, expensive, messy).
Loki Way: Use a recipe card that lists the fruit and the milk separately. (Easy, cheap, precise).

This allows the system to be smaller, train on less data, and perform the "cross-identity" trick (making a stranger act like someone else) without needing special training for it.

Technical Summary: Loki – Representation over Architecture for Diffusion-Based Portrait Animation

Problem Statement

Portrait animation aims to transfer the facial expressions and head pose from a driver video clip onto a single reference image while preserving the reference's identity. The practical challenge lies in cross-identity reenactment, where the driver and reference are different individuals.

Current state-of-the-art diffusion-based systems address this by stacking separate, trained modules for identity, expression, and pose. These systems typically condition on RGB signals (or audio). The paper argues that RGB fundamentally entangles identity, pose, and expression at every pixel. Consequently, the disentanglement performed by these stacked modules is statistical rather than structural, leading to issues such as driver identity leaking into the output and the attenuation of fine-grained expression details toward a generic prior. Furthermore, these complex architectures require massive training datasets and significant parameter counts.

Methodology: Loki

Loki proposes a paradigm shift from architectural complexity to representational choice. Instead of learning to disentangle features from RGB, Loki conditions the diffusion backbone on a parametric face model (FLAME) whose axes for identity, expression, and pose are independent by construction.

1. The Driver Path (Expression and Pose)

Loki rasterizes FLAME parameters into a spatial driver map that serves as the conditioning signal for expression and head pose. This map consists of two distinct channel groups:

Positional Encoding (42 channels): Encodes the canonical template-space coordinates of the FLAME mesh vertices using sinusoidal positional encoding. This identifies which point on the face is visible at each pixel. Crucially, because these values are derived from the template space, they are identity-invariant; the same anatomical point (e.g., the nose tip) always produces the same encoding regardless of the person's identity.
Expression Deformation (3 channels): Encodes the 3D displacement vectors ( $\Delta x, \Delta y, \Delta z$ ) of the mesh vertices relative to a neutral expression. This captures how the face has moved (e.g., lip corners stretching, jaw dropping). Since FLAME's expression basis is orthogonal to its identity basis, these deformations depend solely on the expression coefficients, not the identity or pose.

This driver map is downsampled by a lightweight encoder and injected into the generation UNet. Because the map contains no identity information by construction, the model does not need to learn to separate identity from expression.

2. The Identity Path

Identity is routed separately through the diffusion backbone's own pretrained features. A frozen copy of the Stable Diffusion 2.1 UNet processes the reference image, and its self-attention features are injected into the generation UNet via key-value injection. This leverages the backbone's existing generative prior to maintain identity without requiring a dedicated, trained identity adapter or loss.

3. Cross-Identity Inference via Parametric Substitution

The core mechanism enabling cross-identity reenactment without specific training data is parametric substitution. During inference, the system constructs a driver map using:

The reference's identity shape ( $\vec{\beta}_{ref}$ ) and camera.
The driver's expression ( $\vec{\psi}_{drv}$ ) and pose ( $\vec{\theta}_{drv}$ ).

Because the positional encoding channels are identity-invariant and the deformation channels depend only on expression, the resulting driver map is structurally identical to what the model saw during same-identity training, merely laid out on a different face geometry. The generation UNet performs the same forward pass it was trained on, effectively transferring the driver's motion to the reference's shape.

Key Contributions

Disentangled Parametric Conditioning: A framework that eliminates the need for separate trained modules for expression, pose, and identity. By using a rasterized FLAME representation, expression and pose are delivered via a single spatial signal, while identity is handled by the backbone's prior. This reduces inference parameters by ~43% compared to leading diffusion baselines.
Training Efficiency: The model is trained on 1,496× less video data (approx. 1.7 hours vs. thousands of hours for baselines) and achieves cross-identity capabilities purely through parametric substitution, requiring no cross-identity training pairs.
New Evaluation Metrics: The paper introduces two metrics grounded in the parametric face model to directly measure fidelity to the driver, addressing the gap in existing metrics that fail to measure motion tracking:
- Head Pose Follow (HPF): Measures the geodesic angular distance between the driver's and generated head rotation trajectories.
- Expression Follow (HEF): Measures the pixel-wise difference in expression deformation maps when the driver's expression is substituted into the target's pose and shape.

Results

Evaluated on the HDTF dataset for cross-identity reenactment:

Motion Fidelity: Loki leads or co-leads on the proposed HPF and HEF metrics. It achieves an HEF of 0.0828 (lower is better), outperforming visual-driven baselines like HunyuanPortrait (0.0850) and significantly outperforming audio-driven methods (0.11–0.12).
Qualitative Performance: Loki reproduces large head rotations, wide mouth openings, and asymmetric expressions with higher fidelity than baselines, which tend to under-rotate heads or attenuate expression amplitude toward a moderate prior.
Resource Efficiency: Loki trains on 10,649 clips (approx. 1.7 hours) with 1.77B parameters, whereas leading baselines train on 166,000+ clips (up to 2,543 hours) with 2.18B–3.11B parameters.
Ablation Studies: Removing the rasterization (using raw vectors) or the positional encoding channels significantly degrades cross-identity performance, confirming that the spatial inductive bias and identity-invariant coordinate system are critical.

Significance and Limitations

The paper claims that the architectural complexity of recent portrait animation systems can be replaced by a representational choice: conditioning on a parametric model with independent axes. This approach allows cross-identity reenactment to emerge naturally as a coefficient substitution rather than a learned capability.

However, the authors are modest regarding the system's boundaries:

FLAME Dependency: The conditioning is bounded by the fidelity of the underlying FLAME fits. Expressions outside the model's basis, occluded faces, or fitting failures propagate directly into the driver map.
Pixel Metrics: The system ranks lower on traditional pixel-aligned metrics (PSNR, SSIM, LPIPS) compared to RGB-conditioned baselines. The authors attribute this to the fact that pixel metrics reward systems that preserve ground-truth framing, whereas Loki's faithful driver-following introduces pose and expression variations that pixel metrics penalize.
Future Work: Bridging the gap in pixel metrics without reintroducing identity entanglement is identified as a direction for future research.